Note Compilation: Wang Jiabao
Paper Title: Fast On-device LLM Inference with NPUs (llm.npu)
Paper Link: https://dl.acm.org/doi/abs/10.1145/3669940.3707239
Code Link: https://github.com/UbiquitousLearning/mllm
Published Conference: ASPLOS 2025
Introduction
The biggest challenge in deploying large language models on mobile and edge devices is the slow inference during the prefill stage, which is also very power-consuming. Even for models with around 1 billion parameters, users have to wait several seconds, significantly degrading the experience.
“The first” inference system utilizing the device’s Neural Processing Unit (NPU) to offload and reduce prefill latency is called llm.npu. Through a three-layer optimization framework of Prompt-Tensor-Block, it systematically migrates the prefill computation of LLMs to be executed on mobile NPUs. The results show that prefill acceleration reaches up to 43.6×, with energy consumption reduced by 59.5%, enabling models with around 1 billion parameters to achieve a throughput of >1000 tok/s on mobile devices for the first time.
1. MotivationWith the popularity of large models like ChatGPT and Claude, users hope to use LLMs offline on mobile devices (e.g., AI assistants, translation, knowledge Q&A). However, mobile devices face limitations in scenarios and computing power due to prefill, and this paper aims to accelerate prefill.
However, the two major bottlenecks of edge inference are:
(1) Slow prefill stage: processing hundreds to thousands of tokens at once, with delays in the seconds or even tens of seconds.
(2) High energy consumption and heat generation: prolonged high load leads to overheating of the mobile device and a sharp drop in battery life.
Traditional optimizations (quantization, distillation) can reduce model size but cannot solve the issues of low computational scheduling and hardware utilization from a system level. Therefore, the researchers proposed llm.npu: not only lightweighting the model but also reconstructing the execution process at the system level to maximize the utilization of mobile NPUs.
2. Contributions
This paper proposes a large language model acceleration framework for mobile NPUs, llm.npu, which systematically addresses the issues of high latency and energy consumption during the prefill stage of edge LLM inference. The specific contributions include:
(1) Proposing a prompt-level subgraph reuse mechanism (chunk-sharing graph), which significantly reduces graph construction and scheduling overhead by splitting variable-length contexts into fixed blocks and reusing compiled subgraphs.
(2) Proposing outlier-aware tensor decomposition, which retains only a small number of key activation/weight channels on the CPU/GPU for floating-point computation, while offloading the rest to NPU integer operations, achieving a balance between efficiency and accuracy.
(3) Designing a block-level out-of-order scheduling strategy, which allocates tasks among CPU/GPU/NPU based on operator characteristics and hardware affinity, significantly improving overall hardware utilization.
(4) Achieving up to 43.6× prefill acceleration and a 59.5% reduction in energy consumption on commercial mobile phones, enabling billion-parameter models to reach >1000 tok/s throughput on edge devices for the first time, validating the practical value of the solution.
3. Methodology
Figure 1: Workflow of llm.npu
The llm.npu framework proposed in this paper aims to reconstruct the prefill inference process of large language models to ensure that more computations can run quickly and efficiently on mobile NPUs without significantly compromising accuracy. The entire method can be divided into three core components: prompt-level subgraph reuse, outlier-aware tensor decomposition, and block-level out-of-order scheduling.
(1) Prompt-level Optimization
Figure 2: Explanation of prompt graphs, block graphs, and chunk-sharing graphs, with block length set to 32.
As the first step of the entire method, the authors focus on the overhead of NPU graph compilation and scheduling. In edge inference, the length of the user input context varies dynamically; for example, in a dialogue scenario, the input may be dozens of tokens or over a thousand tokens. For NPUs, each time a new input shape is encountered, it often requires rebuilding and compiling the computation graph, which is both time-consuming and energy-consuming.
Core Idea: To address this issue, the paper proposes the chunk-sharing graph. It splits variable-length input sequences into multiple fixed-size “chunks” and pre-compiles subgraphs for these chunks. During actual inference, these subgraphs can be directly concatenated and reused without regenerating the graph for each input.
a) Input Chunking: Dividing long sequences into fixed-size chunks, e.g., one chunk for every 128 tokens;
b) Subgraph Pre-compilation: Compiling a computation graph for each chunk once and storing it as a reusable module;
c) Graph Concatenation Execution: When user input arrives, concatenating the corresponding subgraphs by length and running them directly on the NPU;
The paper reports that on commercial mobile phones, the time spent on graph construction and scheduling accounts for 15%–25% of the prefill stage latency. Through the chunk-sharing mechanism, the graph preparation time is almost completely eliminated, significantly reducing prefill latency.
(2) Outlier-aware Tensor Decomposition
Figure 3: Workflow of shadow outlier execution.
After reducing the graph preparation overhead, the next challenge is quantization error. For LLMs, low-bit integer computation is essential for edge inference, but there are often a few extreme values (outliers) in weights and activations. If all channels are quantized uniformly, these outliers will be severely compressed, leading to a significant drop in model performance.
Core Idea: The authors propose outlier-aware decomposition. Only a small number of outlier channels retain high-precision floating-point computation, while the vast majority of “normal” channels are offloaded to the NPU for integer execution. This approach maintains overall accuracy while maximizing the utilization of the NPU’s integer computing power.
a) Outlier Detection: Identifying outlier channels that account for about 1%–3% of the total by analyzing the distribution of activations and weights;
b) Mixed Computation: Assigning outlier channels to the CPU or GPU for continued floating-point computation;
c) Integer Execution: Running the remaining 97%–99% of channels on the NPU in low-bit formats such as INT8/INT16.
This strategy’s key idea is: “spending precision costs wisely”—using floating-point numbers only on a few critical channels while allowing the rest to run at full speed on the NPU.
(3) Out-of-order Heterogeneous Scheduling
Figure 4: Out-of-order subgraph execution
After completing graph-level and tensor-level optimizations, the paper further addresses the differences in operator characteristics among different Transformer Blocks. In LLMs, each Block can have vastly different operator types and computational intensities: some operators are computation-intensive and suitable for NPUs, while others are bandwidth-sensitive and rely more on GPUs or CPUs. Forcing all Blocks to execute sequentially on the NPU can create performance bottlenecks.
Core Idea: The authors propose heterogeneous out-of-order scheduling. This method dynamically allocates each Block to the appropriate hardware for execution based on its characteristics while allowing Blocks to run out of strict order within a certain range to avoid a single operator slowing down the overall progress.
a) Hardware Affinity Analysis: Determining whether a Block is more suitable for NPU, GPU, or CPU based on arithmetic intensity, memory access requirements, and precision sensitivity;
b) Dynamic Allocation: Assigning most computation-intensive operators to the NPU, bandwidth-sensitive operators to the GPU, and control logic or small-scale floating-point computations to the CPU;
c) Out-of-order Scheduling: Allowing different Blocks to execute in a non-strict order, reducing hardware idle time and improving pipeline efficiency.
This scheduling method significantly enhances overall hardware utilization. Compared to single hardware execution, heterogeneous scheduling provides an additional acceleration of 1.4–2.3×. Ultimately, on commercial mobile phones, the prefill throughput breaks through 1000 tok/s for the first time, with end-to-end latency improvements of up to 32.8×.
4. Experiments
The optimization methods proposed in the paper for edge LLM inference were primarily evaluated on different NPUs and mobile devices. The experimental setup covered various mainstream large models, including LLaMA series, OPT series, Vicuna, OpenFlamingo-9B, LLaVA-13B, and Mixtral (MOE architecture). Performance metrics focused on prefill stage throughput (tokens/s), median latency, and resource utilization of the NPU/edge devices.
(1) Prefill Performance
Figure 5: Prefill speed and energy consumption at different prompt lengths on different devices
In the experiments, llm.npu tested prefill performance on two mobile devices with different prompt lengths (64, 256, 1024 tokens) (Figures 14 and 15). The results show that llm.npu outperforms all baseline methods in both speed and energy consumption, with the advantage becoming more pronounced as the prompt length increases.
For example, for a 1024-token prompt, llm.npu reduces prefill latency by 1843× on the Redmi K70 Pro compared to LLaMA.cpp-CPU, MNN-CPU, MLC-GPU, and TFLite-GPU; on the Redmi K60 Pro, the improvement is 2169×.
In terms of energy consumption, llm.npu achieves energy savings of 35~59× for long prompts, with short prompts also performing well but slightly lower. These improvements are attributed to shadow outlier execution, efficient tensor MatMul, and out-of-order subgraph scheduling, while fully utilizing the low-power characteristics of the NPU, making edge LLM inference both fast and energy-efficient.
(2) End-to-end Performance
Table 1: End-to-end latency comparison using actual mobile applications on the Redmi K70 Pro
llm.npu was evaluated on three types of edge tasks: UI automation of DroidTask, context-aware email replies of LongBench, and chat summarization of Persona-Chat. The results show that llm.npu significantly reduces inference latency across all tasks, especially with longer prompts, where the acceleration is most pronounced; for example, on LongBench, it is 23~46 times faster than LLaMA.cpp-CPU.
On the Persona-Chat dataset, due to the larger number of output tokens and CPU dependency for decoding, the improvement is slightly lower but still better than most baselines. The experiments indicate that the advantages of llm.npu mainly stem from the efficient prefill stage and the full utilization of the NPU, providing reliable support for deploying large language models on resource-constrained devices like mobile phones.
(3) Inference Accuracy (Inference accuracy)
Table 2: Performance accuracy of LLM on different models
Comparisons were made on five LLM benchmarks between FP16, K-Quant, SmoothQuant, and LLM.Int8(). The results show that llm.npu has negligible accuracy loss, averaging only 1% lower than FP16, significantly outperforming other quantization methods. This is attributed to its dynamic, element-level handling of outliers, which allows for full utilization of the NPU’s computing power while maintaining accuracy.
At the same time, the experiments demonstrated the trade-off between accuracy and generation speed. By adjusting the outlier pruning rate, llm.npu can flexibly balance between generation speed and accuracy: less pruning results in higher accuracy but slower speed, while more pruning leads to faster speed but lower accuracy. This result indicates that llm.npu can be optimized flexibly according to application needs, supporting efficient deployment of large models on edge devices.
5. Conclusion
llm.npu successfully migrates the prefill computation of large language models to edge NPUs through a three-layer optimization strategy of Prompt-Tensor-Block, achieving significant acceleration in prefill inference and end-to-end generation while maintaining nearly lossless accuracy. Combined with dynamic outlier handling and out-of-order subgraph scheduling, llm.npu demonstrates the feasibility and efficiency of deploying large models on mobile devices. Its compatibility with various LLM architectures without complex modifications makes llm.npu a new paradigm for edge AI inference. In the future, this method is expected to integrate with more advanced techniques such as sparsification and dynamic quantization, promoting the deployment and optimization of large models in more intelligent applications.
6. Personal Thoughts
Through the collaborative design of hardware, systems, and models, the bottlenecks of performance and energy efficiency are truly broken at the edge. In my understanding, this is not just an academic optimization but a revelation for the industry: edge inference is transitioning from “feasibility exploration” to “practical implementation.” The three-layer optimization of Prompt-Tensor-Block introduced in this paper also indicates that the future development of edge intelligence must closely integrate algorithm optimization with hardware architecture and system scheduling. This aligns closely with the research direction of Banbu and further convinces me that edge inference will be a key driving force in promoting the next generation of intelligent terminals and AIoT proliferation.