Keywords: V-SEEK, LLM Inference Optimization, RISC-V, SOPHON SG2042, llama.cpp, NUMA Optimization
- V–SEEK: ACCELERATING LLM REASONING ON OPEN-HARDWARE SERVER-CLASS RISC-V PLATFORMS
- https://arxiv.org/abs/2503.17422
In recent years, the exponential growth of large language models (LLMs) has relied on GPU-based systems. However, CPUs are gradually becoming a flexible and cost-effective alternative, especially for inference (the phase where the model has been trained and is only making predictions) and reasoning workloads (tasks that require multi-step logical deductions).
RISC-V (an open-source, royalty-free, customizable instruction set architecture) has rapidly gained attention in this field due to its open and vendor-neutral ISA (Instruction Set Architecture).
However, the RISC-V hardware and its supporting software ecosystem for LLM workloads are not yet fully mature and seamless, as specific domain tuning is required.
This article aims to fill this gap, focusing on optimizing LLM inference on the SOPHON SG2042; the SG2042 is the first commercially available multi-core RISC-V CPU with vector processing capabilities.
On two recently optimized state-of-the-art (SOTA) open-source LLMs—DeepSeek R1 Distill Llama 8B and DeepSeek R1 Distill QWEN 14B—we achieved:
- token generation of 4.32 / 2.29 token/s
- prompt processing throughput of 6.54 / 3.68 token/s, achieving a maximum acceleration of 2.9× / 3.0× compared to our baseline implementation.
Table of Contents
- Table of Contents
- 1. Introduction
- 2. Research Methodology
- 2.1 High-Performance Kernel
- 2.2 Compiler Toolchain
- 2.3 Model Mapping Optimization
- 3. Experimental Results and Analysis
- Kernel Scaling
- Impact of Different Compilers
- Impact of NUMA Strategies
- Performance Summary
- References
1. Introduction
Hyperscalers (e.g., AWS) and AI deployment companies (e.g., OpenAI) typically use GPU clusters or dedicated accelerators (such as TPUs, Tensor Processing Units) to accelerate LLM workloads. However, multi-core CPUs have also been explored recently for LLM acceleration [2], as they provide higher flexibility at a lower hardware cost, especially suitable for on-premise deployments and low-latency edge servers.
Existing research mainly focuses on x86 and ARM, while multi-core chips based on the flexible and open-source RISC-V instruction set architecture have been relatively underexplored [1].
To fill this gap, this work adapts and optimizes the industry-leading LLM inference framework llama.cpp [7] to the first commercially available, general-purpose multi-core RISC-V platform—SOPHON SG2042 [1].
-
On two recently open-sourced models optimized for inference (DeepSeek R1 Distill Llama 8B / QWEN 14B), we achieved up to 3.0× acceleration in token generation and 2.8× in prompt processing (at 4-bit quantization precision), reaching throughputs of 4.32 / 2.29 and 6.54 / 3.68 token/s, respectively.
-
On vanilla Llama 7B, we achieved token generation of 6.63 token/s and prompt processing of 13.07 token/s, which is a 4.3× / 5.5× acceleration compared to the baseline implementation, and a 1.65× improvement over the best results reported on SG2042 [8], while being competitive with mature x86 CPU inference performance.
2. Research Methodology
To explore optimization options for LLM inference on RISC-V server-class platforms, we selected MILK-V Pioneer as the target platform, which features a 64-core SOPHON SG2042 CPU and is equipped with 128 GB of DRAM memory. The platform block diagram is shown in Figure 1-center.
We identified three paths to address the problem, all at the software level, inspired by related work on other architectures [5,6,3]:
2.1 High-Performance Kernel
Develop optimized, quantized computation kernels (a segment of low-level code specifically for matrix operations) for critical LLM layers, fully utilizing hardware resources while considering memory structure, pipeline (the order of instruction execution), and vectorization capabilities.
Figure 1-right presents the pseudocode of our proposed kernel:
- First, quantize the fp32 (32-bit floating point) input (vector or thin matrix) to int8 (8-bit integer);
- Next, execute two nested loops to complete the GEMV (General Matrix-Vector multiplication) operation, where the outer loop iterates over the rows of input matrix A with a step of 2, and the inner loop iterates over its columns with a step of 32.
- After the column loop, perform de-quantization (restoring integers back to floating-point numbers), combining the A block and B’s scale factors to generate the output fp32 values.
This new kernel utilizes the platform’s vector units while optimizing data locality (keeping data close to the compute units to reduce memory access latency).
2.2 Compiler Toolchain
Select an appropriate compiler toolchain that supports advanced optimization passes and can leverage existing ISA extensions.
In our scenario, the kernel is compiled using the Xuantie branch of GCC 10.4 because only this version supports the hardware vector units of the Sophon SG2042. For the entire llama.cpp framework, we considered two alternatives: GCC 13.2 and Clang 19 (Xuantie GCC 10.4 is incompatible with the latest llama.cpp).
2.3 Model Mapping Optimization
Optimize model mapping (the process of allocating model weights and computation tasks to hardware), particularly page/thread allocation, to address the complex memory hierarchy of such systems. Specifically, we explored four strategies with different numactl option combinations to address non-uniform memory access (NUMA) latency:
- NUMA Balancing enabled, all other options disabled;
- All options disabled;
- Balancing disabled + Core Binding enabled;
- Balancing disabled + Memory Interleaving enabled.
We applied the above optimizations to the llama.cpp [7] framework and tested on three progressively larger open-source LLMs, all using Q4_0 quantization (vanilla Llama 7B, DeepSeek R1 Distill Llama 8B, DeepSeek R1 Distill QWEN 14B, referred to as 7B, 8B, and 14B, respectively).
3. Experimental Results and Analysis
To demonstrate the optimization effects, we executed a prefill with the user prompt “Explain to me what is RISC-V, what are its principles and why it is so cool?” (22 tokens in total) on the three LLMs, while taking the average of 256 test-generated tokens for token generation performance.
Kernel Scaling
Figure 2 compares the single-thread scalability of multiple baseline kernels (the GGML and OpenBLAS default implementations that come with llama.cpp) with our proposed kernel.
Compared to the best baseline, we achieved an average increase of 38.3% in GOPS (Giga Operations Per Second), with a peak improvement of 56.3% at a matrix size of 4096.
Impact of Different Compilers
Figure 3 evaluates the inference performance of the DeepSeek 8B model when compiled with Clang or GCC, both using our proposed kernel.
Clang 19 consistently outperformed GCC 13.2, with an average performance improvement of 34% in token generation and 25% in prefill. The key reason is Clang’s support for ISA extensions and more advanced compilation optimizations (such as more aggressive inlining and loop unrolling). Regardless of the compiler used, performance degradation occurs when the number of threads exceeds 32. This behavior is attributed to the default NUMA balancing strategy, which is not ideal for predictable loads like LLM inference, leading to significant thread and memory page migrations.
Impact of NUMA Strategies
Indeed, after disabling NUMA balancing and enabling memory interleaving, we achieved the best results of 4.32 token/s for token generation and 6.54 token/s for prefill at 64 threads, which benefited from a significant reduction in memory page migrations.
Performance Summary
Thanks to our optimizations, the three LLMs—7B, 8B, and 14B—achieved maximum throughputs of 13.07 / 6.54 / 3.68 token/s, with maximum improvements of 5.5× / 2.9× / 3× compared to the baseline llama.cpp.
- Compared to the best results reported on SG2042 [8], we achieved a peak throughput improvement of 1.65× on Llama 7B.
- Compared to a similar and more mature x86 platform—the 64-core AMD EPYC 7742—we improved energy efficiency by 1.2× (55 token/s/mW vs. 45 token/s/mW).
References
