Analysis: Inference Deployment of Large Language Models on RISC-V Servers

The application of large language models in server environments during the AI era, especially in the inference phase, presents numerous opportunities. AI can be divided into two main stages: training and inference. From pre-training to fine-tuning and reinforcement, a general large model is developed, which is then used to generate application models; the inference side constructs intelligent agents based on application models to complete various task inferences, forming a closed loop.

On July 17, at the fifth RISC-V China Summit, Shi Yijun, Deputy General Manager of ZTE Microelectronics, delivered a presentation titled “Research on Inference Deployment of Large Language Models on RISC-V Servers and Architecture,” where he delved into the current status, challenges, and opportunities of deploying large language models for inference on RISC-V servers.

Shi emphasized that inference centered around large models is currently a key focus for implementation, and optimizing power consumption, performance, and area (PPA) for this purpose holds long-term value, warranting collective investment from the industry.

Shi analyzed the evolution direction of large models: in terms of algorithm architecture, there is a shift from traditional Dense models to Mixture of Experts (MoE) models, and the emergence of DeepSeek has further propelled the transition from power-oriented to efficiency-oriented computing. Under this new architecture, CPUs, including RISC-V, have the opportunity to provide critical support for inference deployment, with cost reduction and efficiency enhancement as core objectives.

Shi summarized three major characteristics of large model development:

1. The Continuous Impact of Algorithms and Scaling LawsOnce model parameters exceed trillions, significant sparsity will be observed, with the number of parameters required for each inference decreasing by an order of magnitude, leading to reduced computing power demands, while cache, bandwidth, and storage capacity requirements continue to rise.
2. Storage as the CoreMore than half of AI solutions revolve around storage, which is a significant trend in large model architecture.

When optimizing total cost of ownership (TCO), large model inference faces two core contradictions:Contradiction One: The Switching Differences in Driving Factors of Computing and Bandwidth

• Prefill phase (first token generation): requires meeting the time requirements for the first token, relying on high computing power;
• Decode phase (subsequent token generation): requires certain bandwidth to ensure task throughput.

Contradiction Two: The Divergence Between Storage Costs and Data Activity LevelsIn MoE models, only a few parameters are activated for each token. Statistics show that frequently used parameters account for only a small portion, while low-frequency parameters still incur high storage costs.

Based on the above analysis, Shi believes there is significant optimization potential in large model inference. The horizontal axis in the diagram represents data concurrency, while the vertical axis represents computing power, bandwidth, and capacity requirements:

• Training scenarios are located in the upper right corner, requiring high data distribution, computing power, bandwidth, and capacity;
• Inference scenarios have broader requirements and may need various architecture adaptations.

Specifically:

• Low Concurrency Scenarios (Edge/On-device)CPU or pure CPU solutions offer better cost-performance ratios (DDR storage costs are much lower than HBM), making them a focus for small and medium enterprises and private domain inference applications;
• High Concurrency Scenarios (Cloud-side Large-scale Deployment)Pure GPU costs are high; if combined with CPU to offload some tasks, the number of GPUs required can be reduced, optimizing overall TCO.

“Hybrid inference is the future trend,” Shi pointed out. Under computational sparsity, CPUs can share the attention computation during the decode phase, improving communication efficiency and reducing deployment costs.

Despite the opportunities for RISC-V in AI inference, its practical application still faces multiple challenges:

Challenge One: Model Architecture and Computing Power Adaptation

• Algorithm Optimization SpaceCurrently, MoE is mainstream, but the industry is still exploring better algorithms, and the underlying computing power must support efficient computation to enhance server computing power utilization;
• Storage ManagementLarge models require long context support (a key demand for intelligent agents), making storage management crucial;
• Multi-core Communication and Heterogeneous CoordinationMulti-core communication efficiency and heterogeneous computing coordination still need optimization.

Challenge Two: Computing Power and Data Format Adaptation

• Low Bit Precision SupportLarge models have an urgent need for low bit precision (which can significantly reduce bandwidth, computing power, and storage requirements), but existing architectures face difficulties operating on low-byte data;
• Data Scaling MechanismEfficient support for data scaling is needed to maintain precision while reducing bandwidth/computing power; the openness and flexibility of RISC-V can quickly adapt to new demands, optimizing large model deployment performance.

Challenge Three: Ultra-long Context Support

• Storage ConsumptionUltra-long contexts require storing historical information, placing immense pressure on storage. CPUs are inherently strong in storage management (GPU deployments also rely on CPUs), and if necessary computing power is added to CPUs to share attention computation, it can optimize long context processing and reduce TCO.

Challenge Four: Multi-core Communication Efficiency

• Overhead of Multi-core ExpansionDuring multi-core deployment, communication efficiency is a significant bottleneck. In one case, multi-core communication accounted for 39% of the overhead, necessitating targeted optimizations through software improvements and hardware-level mechanisms (even RISC-V instruction set architecture ISA standards); communication efficiency is crucial for system performance enhancement during multi-die and multi-chip expansions.

Despite facing numerous challenges, RISC-V still possesses undeniable potential: its instruction set’s openness is one of its core advantages, allowing everyone to participate, greatly fostering innovation without restrictions; customizability supports various fields in tailoring and trial-and-error based on needs, leading to standardization after convergence, thus accelerating the realization of advantages; an open ecosystem is vital for AI applications, enabling rapid construction of toolchains and accelerating innovation iterations; additionally, vector extensions (RVV) have a solid foundation, providing support for future development.

Shi believes that to promote the deployment of large model inference on RISC-V servers, efforts must focus on standardization and ecosystem co-construction. On one hand, the issue of standardization for interfacing with GPUs must be resolved—existing models are primarily developed based on CPUs, and when offloading tasks to GPUs, differences in data formats, algorithm details, etc., can lead to migration difficulties, thus requiring optimization at the standard level to reduce migration challenges; on the other hand, foundational architecture standards must be improved, including matrix expansion, computing power data formats, and multi-core communication efficiency, to release more development opportunities through unified foundational technology standards.

From a positioning and path perspective, RISC-V should focus on two major directions in AI scenarios: first, leveraging small and medium CPUs to play a lightweight inference advantage in light tasks and private domain scenarios; second, collaborating with GPUs in a hybrid deployment ecosystem to optimize the overall TCO of the existing ecosystem.

At the same time, innovation in foundational architecture and ecosystem innovation must be advanced simultaneously—unifying foundational computing power standards to support scenario adaptation; reducing AI total costs through CPU and GPU collaboration; and the industry must jointly build a technological and industrial ecosystem, from architectural innovation forming standards (as the cornerstone of development) to the community and industry creating usable and effective products, ultimately achieving end-to-end application in scenarios and realizing industrial demonstration deployment.

Shi emphasized that the development of RISC-V is a collective endeavor of the industry chain. Through industrial collaboration, gathering talents from all sides, and jointly building a sustainable RISC-V ecosystem, we can promote its more solid steps in the field of AI inference.