Qualcomm’s On-Device Generative AI: Innovations in Hexagon NPU Architecture Achieving 98% Performance Improvement, 0.6 Seconds Image Generation, and 20 Tokens/s LLM Inference

Keywords: on-device generative AI, NPU (Neural Processing Unit), heterogeneous computing, Qualcomm AI Engine, Snapdragon, AI software stack

• Unlocking on-device generative AI with an NPU and heterogeneous computing
• https://www.qualcomm.com/content/dam/qcomm-martech/dm-assets/documents/Unlocking-on-device-generative-AI-with-an-NPU-and-heterogeneous-computing.pdf
• https://www.qualcomm.com/developer/artificial-intelligence
• This article has 13,299 words and takes 43 minutes to read, podcast 8 minutes

Related ArticlesEvaluating the LLM inference performance and energy efficiency of Qualcomm 100 Ultra acceleration card in HPC clusters2.1x acceleration! Throughput exceeds equivalent Qwen3 by 17%! Huawei Pangu Light: Acceleration under aggressive pruning! Ascend NPU achieves a balance of accuracy and efficiency!HeteroLLM: Achieving NPU-GPU parallel heterogeneous LLM inference using mobile SoC! Taking Qualcomm 8 Gen 3’s NPU GPU as an example

The wave of generative AI is sweeping across various industries, but diverse application demands and stringent device power consumption and heat dissipation limits have caused general computing architectures to hit a “performance bottleneck”— users expect instant responses for AI creation and continuous intelligent interaction, yet traditional CPUs and GPUs struggle to balance efficiency and low consumption. How should this “on-device AI breakthrough battle” be resolved?

Qualcomm provides the core answer in “Unlocking on-device generative AI with an NPU and heterogeneous computing”: a new architecture centered on “custom NPU + heterogeneous computing collaboration” to reconstruct device-side AI capabilities.

The Hexagon NPU is designed from the ground up specifically for generative AI, evolving through multiple generations—from supporting basic audio AI in 2015 to adapting to large models with billions of parameters in 2023, achieving “sustained high-performance inference at low power consumption” through scalar/vector/tensor accelerator integration, INT4 hardware acceleration, dedicated power rails, and other innovations, with the Snapdragon 8 Gen 3’s NPU achieving a 98% performance improvement and a 40% optimization in performance per watt.

The real breakthrough lies in “heterogeneous computing“: deeply coordinating the NPU with Adreno GPU, Kryo/Oryon CPU, Sensing Hub, and other processors, allowing each hardware to “play its role”—the CPU handles low-latency small models, the GPU processes high-precision parallel tasks, and the NPU takes on the core AI load, even enabling the Avatar AI assistant to simultaneously run voice recognition (Sensing Hub), large language models (NPU), and image rendering (GPU), achieving “seamless multitasking“.

More importantly, this is not just a laboratory technology:

• The Snapdragon 8 Gen 3 phone can generate a 512×512 image in 0.6 seconds, with the Llama 2-7B model generating 20 tokens per second;
• The NPU performance of the Snapdragon X Elite PC far exceeds that of competing X86 chips, achieving an AI benchmark score 8.6 times that of its competitors.

Additionally, the Qualcomm AI Stack provides the capability of “one-time development and multi-end deployment,” allowing developers to easily access full-stack AI resources—this “on-device generative AI revolution” has already transitioned from concept to practicality.

unsetunsetKey Issuesunsetunset

Question 1. Although Qualcomm’s Hexagon NPU emphasizes differentiated designs such as “custom ISA + microchip inference” and achieves a 98% performance improvement with Snapdragon 8 Gen 3, how does its hardware acceleration module’s energy efficiency ratio (tokens processed per watt/image generation speed) specifically compare to competing NPUs from Apple, MediaTek, etc. when handling 10B+ parameter multimodal models (e.g., LLM + LVM collaboration)? The author does not mention cross-vendor quantification comparisons under the same model; is this difference sufficient to support the experience barrier for on-device generative AI?

Energy efficiency ratio and experience barrier of Hexagon NPU compared to competitors in 10B+ multimodal models. Although no direct values are provided for the energy efficiency ratio of Hexagon NPU compared to competitors like Apple and MediaTek under “10B+ parameter multimodal models (LLM + LVM collaboration),” nor is there cross-vendor quantification comparison data mentioned, it can be inferred through the disclosed technical features and performance benchmarks that it indirectly supports its differentiated advantages and experience barriers:

1. Hardware and energy efficiency optimization technology support: The Hexagon NPU achieves core optimization through “custom ISA (Instruction Set Architecture) + microchip inference (reducing memory traffic by 10+ layers) + dedicated power rails + native INT4 support”—the Snapdragon 8 Gen 3’s Hexagon NPU not only achieves a 98% performance improvement, but also achieves a 40% performance optimization per watt, with INT4 quantization improving performance by 90% and efficiency by 60% compared to INT8, directly addressing the “high memory usage and high computational demand” pain points of multimodal models, reducing energy loss at the hardware level;
2. Actual application and benchmark testing lead: In 7B parameter models (such as Llama 2-7B, Stable Diffusion), the Hexagon NPU has already shown clear advantages:
1. Mobile Llama 2-7B reaches 20 tokens/s, Stable Diffusion generates 512×512 images in < 0.6 seconds,
2. PC (Snapdragon X Elite) Llama 2-7B reaches 30 tokens/s;
3. In benchmark tests, the Snapdragon 8 Gen 3’s MobileBert model performance exceeds competitors by 321%, and the Snapdragon X Elite in UL Procyon AI Benchmark exceeds X86 competitors by 8.6 times, reflecting its leading efficiency in model processing, which can extend to support potential energy efficiency advantages for 10B+ models;
3. Core logic of experience barriers: The Hexagon NPU is not an isolated hardware but is integrated into Qualcomm AI Engine’s heterogeneous system (in collaboration with Adreno GPU, Oryon CPU), capable of splitting tasks for multimodal models’ “text generation (NPU excels) + image rendering (GPU excels) + low-latency control (CPU excels),” avoiding performance waste from a single processor’s “full-scene adaptation”—this system-level collaborative capability is difficult for competitors to replicate, ultimately translating into a user-perceived “low latency + long battery life” experience barrier.

Question 2. The author points out that Qualcomm AI Engine allocates multi-processor tasks through heterogeneous computing (e.g., the Avatar assistant uses NPU for LLM, GPU for rendering the avatar), but the data transfer between different processors (e.g., TTS data from NPU to CPU, rendering data from CPU to GPU) relies on the memory subsystem. In concurrent scenarios of real-time generation (20 tokens/s) + image rendering for 7B parameter LLM, what percentage of total time does data transfer latency account for? Is there a situation where the heterogeneous collaborative energy efficiency ratio is lower than that of a “single high-performance NPU” due to transfer bottlenecks?

On the data transfer latency ratio in heterogeneous computing and its energy efficiency comparison with a single NPU, the author does not provide the “specific percentage of data transfer latency in total time”.

However, based on the disclosed details of the heterogeneous architecture design, it can be inferred that the transfer bottleneck has been significantly alleviated, and the energy efficiency of heterogeneous collaboration is superior to that of a single high-performance NPU:

1. Hardware foundation for data transfer optimization: Qualcomm AI Engine’s heterogeneous system emphasizes “on-chip memory sharing + bandwidth enhancement”—the Snapdragon 8 Gen 3’s Hexagon NPU doubles the shared memory bandwidth, and all processors are integrated into the same SoC (System on Chip), avoiding the high latency and power consumption of “I/O transfer between discrete chips”; the author explicitly mentions in the Avatar AI assistant case that “data is transmitted on-chip as much as possible to reduce cross-chip interaction,” while microchip inference technology “eliminates memory traffic of over 10 layers of neural networks,” reducing data transfer demand from the source;
2. Energy efficiency advantage logic of heterogeneous collaboration: While a single NPU excels at AI inference, it cannot efficiently handle multi-task scenarios (e.g., LLM generation + image rendering)—for example, in the Avatar case, the NPU runs Llama 2-7B (core AI task), the GPU handles avatar rendering (graphics parallel task), and the CPU processes TTS (low-latency control task), allowing each processor to leverage its “specialized advantages”: if a single NPU undertakes rendering tasks, it would sacrifice AI inference performance (NPU’s graphics processing capability is weaker than that of the GPU); if the GPU undertakes AI tasks, it would significantly increase power consumption (GPU’s general computing efficiency is lower than that of NPU). The author mentions that the Snapdragon 8 Gen 3’s “40% performance optimization per watt” is based on heterogeneous collaboration, indicating that its overall energy efficiency is superior to that of a single processor solution;
3. Core design for latency control: Qualcomm AI Engine’s software stack (such as Qualcomm AI Stack) further reduces transfer latency through “task pre-scheduling + memory pre-allocation,” ensuring that text data generated by the NPU can be seamlessly passed to the CPU for TTS and to the GPU for rendering. The demonstration of the Avatar assistant’s “real-time voice interaction + synchronized avatar animation” also indirectly proves that transfer latency has not significantly impacted user experience.

Question 3. The author mentions that INT4 quantization can reduce memory usage, and the Snapdragon 8 Gen 3 supports LPDDR5x 77GB/s bandwidth, but facing a 70B parameter LLM (approximately 35GB memory after INT4 quantization), the current maximum memory capacity of the Snapdragon platform (about 16GB for mainstream flagship phones) is far from sufficient. If relying on cloud-side sharded inference, it would lose the privacy advantage of on-device processing. Does Qualcomm have a clear roadmap for hardware memory expansion (such as stacked storage) or model compression (such as sparsity + quantization combination) to address the on-device memory bottleneck for 70B+ large models?

On the on-device memory bottleneck for 70B parameter LLM and Qualcomm’s solution roadmap, the author does not explicitly mention a specific roadmap for “stacked storage” or “sparsity + quantization combination,” but has disclosedshort-term technical directions to alleviate the bottleneck, and clearly focuses on a full-stack strategy of “model optimization + hardware bandwidth enhancement” in the long term:

1. Current core means to alleviate memory pressure:

• INT4 quantization compression: The Hexagon NPU natively supports INT4 integer precision, reducing model memory usage to half of INT8 and a quarter of FP16—after INT4 quantization, the memory requirement for a 70B parameter model can be reduced from 140GB (70B×2Byte) to 35GB. Although this is still higher than the current flagship phone’s 16GB memory, the gap has been significantly narrowed; the author mentions that the “AIMET toolkit (Qualcomm AI Model Efficiency Toolkit)” reduces precision loss through “quantization-aware training” to ensure the usability of the compressed model;
• Memory bandwidth enhancement: The current Snapdragon platform supports LPDDR5x 4.8GHz (77GB/s bandwidth), and the author clearly states that this configuration is to “address the growing memory demands of generative AI,” as high bandwidth can reduce “memory read wait times,” indirectly alleviating the memory access pressure of large models;

Long-term technical direction implications:

• Model level: The author emphasizes “full-stack AI optimization,” including quantization, compression, neural architecture search (NAS), conditional computation, etc., inferring that in the future, it will further combine “sparsity (trimming redundant parameters) + INT4 quantization” to compress the 70B model to a scale more suitable for on-device memory (e.g., within 16GB);
• Hardware level: The author does not mention stacked storage, but the continuous upgrade of the NPU memory subsystem (e.g., Snapdragon 8 Gen 3’s shared bandwidth doubling) implies that in the future, it will enhance memory capacity and efficiency through “higher specification memory (e.g., LPDDR6) + larger on-chip cache”;
• Implementation priority: Currently focusing on “on-device implementation of 7B-10B parameter models” (e.g., Llama 2-7B, Stable Diffusion), indicating that Qualcomm is taking a “gradual approach”—first achieving large-scale applications of medium and small models, then gradually breaking through the memory bottleneck of 70B+ large models through technological iteration, while avoiding the risk of “sacrificing current experience for large models.”;

unsetunsetArticle Directoryunsetunset

• Key Issues
• Article Directory
• 1. Qualcomm Hexagon NPU: From “Adapting AI” to “Defining AI” Design Thinking
• 1.1 Evolution of NPU: From Single Task to Multimodal Generation Capability Leap
• 1.2 Core Design Thinking of NPU: System-Level Perspective and Full-Stack Optimization
• 1.3 Key Strategies to Break Bottlenecks: Shifting Focus from “Computational Bottlenecks” to “Memory Bottlenecks”
• 2. Qualcomm AI Software Technology Stack: The “Bridge” Connecting Hardware and Developers
• 2.1 Overall Architecture of AI Software Stack: Four Layers of Collaboration from “Framework Layer” to “Hardware Layer”
• 2.2 Upper Layer: Framework and Runtime Support—Lowering Development Barriers
• 2.3 Middle Layer: Development Tools and Optimization Layer—Releasing Hardware Performance
• 2.4 Lower Layer: System Software and Driver Layer—The “Link” Connecting Hardware and Software
• 2.5 Core Value: “One Development, Multi-End Deployment” and Scalable Empowerment
• 3. Performance: Comprehensive Leadership from Benchmark Testing to Actual Applications
• 3.1 Mobile End: AI Performance Breakthrough of Snapdragon 8 Gen 3
• 3.2 PC End: AI Performance Revolution of Snapdragon X Elite
• 3.3 Practical Cases of Heterogeneous Computing: Synergistic Effects of Avatar AI Assistant
• 4. Conclusion: Core Competitiveness and Future Direction of Qualcomm’s AI Technology System

To join the discussion group, please reply in the NeuralTalk public account background: Let’s be friends

unsetunset1. Qualcomm Hexagon NPU: From “Adapting AI” to “Defining AI” Design Thinkingunsetunset

The NPU is the core carrier of on-device AI computing power, but not all NPUs can meet the demands of generative AI—large language models (LLMs) and large vision models (LVMs) not only require high computing power but also impose stringent requirements on memory bandwidth, energy efficiency, and sustained computing capability.

From its inception, Qualcomm’s Hexagon NPU has followed the design philosophy of “optimizing from a system-level perspective centered on AI workloads,” evolving over more than a decade from the early DSP (Digital Signal Processor) foundation to a high-performance computing unit specifically tailored for generative AI. Its design thinking can be summarized as three core logics: “evolving to meet demands, architecting to break bottlenecks, and system collaboration to enhance efficiency“.

1.1 Evolution of NPU: From Single Task to Multimodal Generation Capability Leap

The evolution of the Hexagon NPU is not an isolated hardware upgrade but is deeply tied to the development of AI models and application scenarios. Qualcomm continuously tracks AI industry trends to ensure that the NPU architecture remains in sync with mainstream model demands, avoiding the dilemma of “hardware being ahead of uselessness and capability lagging behind demand.”

Its evolution can be divided into four key stages, with design decisions at each stage reflecting precise judgments of AI demands at the time:

Foundation Laying Period (2007-2014): Power Reserve Centered on DSP

In 2007, Qualcomm launched the first Hexagon DSP (Digital Signal Processor) on the Snapdragon platform. Although its scalar architecture was not directly positioned for “AI,” it laid a critical foundation for subsequent NPUs—core operations of AI models (such as multiply-accumulate operations in neural network layers) can essentially be decomposed into scalar and vector operations, and the low power consumption and high parallel characteristics of DSP are well-suited for these tasks.

During this stage, Qualcomm’s core thinking was to “lay out general computing capabilities in advance to reserve a technological base for the AI era,” avoiding the need to build hardware from scratch when AI demand exploded.

AI Enlightenment Period (2015-2017): First Integration of AI Engine, Adapting Basic Perception Tasks

In 2015, the Snapdragon 820 processor was launched, integrating the “Qualcomm AI Engine” for the first time, coordinating the Hexagon DSP with the image signal processor (ISP) and CPU to support basic AI tasks such as imaging, audio, and sensors (e.g., photo denoising, voice wake-up).

At this time, AI models were primarily based on simple convolutional neural networks (CNNs), with computational demands concentrated at the scalar and vector levels. The Hexagon DSP, through optimized instruction sets, was already able to meet the needs of this type of “lightweight AI.”

Qualcomm’s design thinking was to “not build a separate NPU, but to quickly enter the AI field through existing processors while accumulating actual workload data.”

AI Acceleration Period (2018-2021): Addition of Tensor Accelerators to Address Complex Models

In 2018, the Hexagon NPU of the Snapdragon 855 processor first introduced the “Hexagon Tensor Accelerator,” a decision directly stemming from the upgrade of AI models—after 2016, AI tasks related to photography and video exploded, and models such as transformers, recurrent neural networks (RNNs), and long short-term memory networks (LSTMs) began to proliferate, requiring a large number of tensor operations (e.g., matrix multiplication), while traditional scalar/vector accelerators were extremely inefficient.

Qualcomm’s core thinking during this stage was to “targetedly solve computational bottlenecks”: the tensor accelerator enhances computational efficiency by several orders of magnitude through hardware-level support for matrix multiplication operations.

For example, an N×N matrix multiplication requires reading 2N² data and executing N³ operations on a scalar accelerator, while the tensor accelerator, through a “compute-intensive design,” improves the “compute operation/memory access ratio” from scalar’s 1:1 to N:1, significantly reducing memory bandwidth usage and energy consumption. At the same time, Qualcomm configured large-capacity shared memory for the NPU, further reducing data transfer losses between processors; this design philosophy remains a core advantage of the Hexagon NPU to this day.

Generative AI Customization Period (2022-Present): Comprehensive Optimization to Adapt to LLM/LVM Demands

After 2022, generative AI exploded, with LLMs (such as Meta Llama 2-7B) and LVMs (such as Stable Diffusion) scaling from “millions” to “billions” and even “hundreds of billions” of parameters, bringing three new challenges: memory bandwidth bottlenecks (the speed of data reading determines performance when generating tokens for LLMs), sustained computing efficiency (long-running models need to control heat), and multimodal collaboration (text, voice, and images require cross-modal processing).

To address these challenges, Qualcomm has made “disruptive upgrades” to the Hexagon NPU on the Snapdragon 8 Gen 2 and 8 Gen 3 platforms, with each improvement reflecting a deep understanding of generative AI demands:

• 2022 (Snapdragon 8 Gen 2): Breaking Through Energy Efficiency and Accuracy Balance introduced “native 4-bit integer (INT4) support,” a key decision by Qualcomm to address LLM memory bottlenecks. The weight parameters of LLMs can maintain high accuracy at 8-bit (INT8) or even 4-bit (INT4) precision, but memory usage and bandwidth requirements can be reduced by 50%-75%. The Hexagon NPU, through hardware-level INT4 acceleration, not only doubles the tensor throughput of INT4 layers but also reduces energy consumption by 60% (compared to INT8). At the same time, the “Microtile Inferencing” technology was introduced: breaking down large neural networks into small “microtiles” that execute independently within the NPU, reducing memory interactions for networks with more than 10 layers, further alleviating bandwidth pressure; a dedicated hardware accelerator for the “multi-head attention mechanism” of transformer models was added, increasing the inference speed of the MobileBERT model by 4.35 times, directly addressing the core computational bottlenecks of generative AI.

• 2023 (Snapdragon 8 Gen 3): Fully Adapting to Generative AI

This generation of Hexagon NPU is defined as a “generative AI dedicated processor,” with core improvements focused on “enhancing sustained performance, optimizing memory efficiency, and increasing computational elasticity”:

1. Dual Breakthroughs in Performance and Energy Efficiency: Compared to the previous generation, AI inference performance improved by 98%, and performance per watt improved by 40%, supporting 7 billion parameter models to run locally for extended periods;
2. Memory Bandwidth Optimization: Shared memory bandwidth doubled, supporting the industry’s fastest LPDDR5x memory (4.8GHz, 77GB/s), directly addressing the memory bottlenecks of LLMs—previously, the speed of token generation for LLMs was limited by memory reading, while high-bandwidth memory increased the token generation speed of Llama 2-7B to 20 tokens/s;
3. Elastic Power Scheduling: A dedicated power rail was added for the Hexagon Tensor Accelerator, allowing dynamic power allocation based on the model’s needs for “scalar/vector/tensor,” avoiding computational waste;
4. Microtile Inference Upgrade: Further optimized the “microtile” splitting logic to adapt to larger-scale LLMs and LVMs, for example, reducing the time for Stable Diffusion to generate 512×512 images to 0.6 seconds, with CLIP (image accuracy metric) close to baseline models.

1.2 Core Design Thinking of NPU: System-Level Perspective and Full-Stack Optimization

The differentiated advantage of Qualcomm’s Hexagon NPU does not solely rely on hardware parameters but stems from a system-level thinking of “from application to hardware.” This thinking can be summarized into three core principles, which are the underlying logic of Qualcomm’s NPU design:

Logic 1: “Not Making an Isolated NPU,” Heterogeneous Collaboration is Key

Qualcomm has always believed that maximizing efficiency for on-device AI cannot rely on a single processor—the CPU excels at low-latency, small model tasks (e.g., text completion), the GPU excels at parallel high-precision computations (e.g., image rendering), the Sensing Hub excels at low-power always-on tasks (e.g., voice wake-up), while the NPU focuses on large-scale AI computations. Therefore, the Hexagon NPU has been integrated into the “Qualcomm AI Engine” heterogeneous architecture from its design inception, collaborating deeply with other processors rather than existing in isolation.

For example, in the AI assistant scenario, the Sensing Hub is responsible for real-time listening to voice commands (power consumption < 1mA), the NPU runs LLM to generate responses, the CPU handles text-to-speech (TTS), and the GPU renders the virtual avatar—this division of labor allows the overall power consumption to be reduced by over 40% compared to running all tasks on a single NPU. Qualcomm’s thinking is: “The value of the NPU lies not in ‘replacing other processors’ but in ‘completing the computational puzzle’ to achieve an effect of ‘1+1>2’ through heterogeneous collaboration.”

Logic 2: “Custom ISA + Rapid Iteration,” Keeping Up with AI Model Trends

Unlike many vendors that rely on third-party IP, Qualcomm independently designs the instruction set architecture (ISA) for the Hexagon NPU. The core advantage of this decision is “rapid adaptation to new models.” The evolution speed of AI models is extremely fast (from CNN to transformers in just 5 years); if relying on third-party ISA, hardware upgrades may have to wait for IP vendors to iterate, potentially lagging by 1-2 years; whereas an independent ISA allows Qualcomm to add dedicated instructions (such as attention mechanism instructions for transformers, INT4 quantization instructions) based on model needs within months.

For example, after LLMs became mainstream in 2023, Qualcomm modified the ISA to add “LLM-specific instructions” in the Hexagon NPU, improving the inference efficiency of Llama 2-7B by 30%, and this process took only 3 months. This rapid iteration capability of “hardware following model trends” is key to Hexagon NPU’s continued leadership.

Logic 3: “Full-Stack Optimization,” A Closed Loop from Model to Hardware

Qualcomm’s NPU design is not “hardware-first” but rather a full-stack collaboration of “model-software-hardware.” The process is:

• First, the AI research team analyzes the computational characteristics of mainstream models (such as LLMs, LVMs) to identify bottlenecks (such as memory bandwidth, attention mechanisms);
• Then, software tools (such as AIMET model optimization tools) validate optimization solutions (such as INT4 quantization);
• Finally, the validated solutions are transformed into hardware designs (such as INT4 accelerators). This “from application to hardware” closed loop ensures that every improvement of the NPU can solve practical problems rather than just piling up parameters.

For example, Qualcomm AI Research found that the memory bottleneck of LLMs mainly stems from “weight reading,” so it developed a “weight compression algorithm” at the software level and designed “high-bandwidth shared memory” at the hardware level, with both working together to reduce the memory usage of LLMs by 50% while keeping precision loss < 1%. This full-stack optimization thinking is the core reason for Hexagon NPU’s leading energy efficiency ratio.

1.3 Key Strategies to Break Bottlenecks: Shifting Focus from “Computational Bottlenecks” to “Memory Bottlenecks”

At different stages of AI development, the core bottlenecks of the NPU differ—early on (2015-2018), it was the “computational bottleneck” (insufficient computing power), in the mid-stage (2019-2021), it was the “energy efficiency bottleneck” (sufficient computing power but high power consumption), and currently (2022-present), it is the “memory bottleneck” (sufficient computing power but slow memory reading). Qualcomm continuously breaks core bottlenecks by dynamically adjusting the design focus of the NPU:

Focus 1: Solving Computational Bottlenecks: Tensor Accelerators + Dedicated Hardware

Before 2018, AI models were primarily based on CNNs, with computational demands concentrated on convolution operations, and traditional scalar/vector accelerators were inefficient. Qualcomm’s solution was “hardware-level acceleration for convolution and tensor operations”: adding tensor accelerators to the Hexagon NPU to directly support matrix multiplication operations, significantly improving computational efficiency by over 10 times; at the same time, dedicated hardware was designed for commonly used activation functions (such as ReLU, Sigmoid) to avoid delays caused by software simulation.

Focus 2: Solving Energy Efficiency Bottlenecks: Fusion Architecture + Dynamic Power Supply

After 2019, as the complexity of AI models increased, the NPU needed to run for extended periods, making energy consumption critical. Qualcomm’s solution was “fusion architecture + dynamic power supply”: in 2020, scalar, vector, and tensor accelerators were fused into a single computing unit, reducing data transfer losses between processors, improving energy efficiency by 30%; in 2022, an independent power rail was added to the NPU, allowing dynamic adjustments of voltage and frequency based on workload, such as reducing power supply when running small models and increasing it when running large models, avoiding energy waste from “overpowering small tasks.”

Focus 3: Solving Memory Bottlenecks: High-Bandwidth Memory + Microtile Inference

Currently, the parameter scale of LLMs/LVMs reaches billions, and the speed of memory reading determines performance. Qualcomm’s solution is “three-layer optimization”:

1. Hardware Layer: Enhancing shared memory bandwidth (doubled in Snapdragon 8 Gen 3), supporting LPDDR5x high-bandwidth memory to reduce data read latency;
2. Software Layer: Using “Microtile Inferencing” to split models into microtiles, completing computations within the NPU to reduce interactions with external memory;
3. Model Layer: Reducing parameter volume through INT4 quantization, lowering memory usage by 75%, for example, the parameters of Llama 2-7B drop from 28GB (FP32) to 3.5GB (INT4), fitting entirely within the NPU’s shared memory, avoiding frequent reads from external memory.

unsetunset2. Qualcomm AI Software Technology Stack: The “Bridge” Connecting Hardware and Developersunsetunset

Having powerful NPU hardware alone is insufficient to drive the large-scale deployment of on-device AI—developers need simple and easy-to-use tools to deploy AI models across different devices (phones, PCs, cars) while fully leveraging hardware performance.

Qualcomm’s solution is the “Qualcomm AI Stack,” a full-stack software system covering the entire process of “model development-optimization-deployment-running,” with the core goal of “allowing developers to write code once and deploy it across all Qualcomm platform devices,” lowering the development threshold for on-device AI.

2.1 Overall Architecture of AI Software Stack: Four Layers of Collaboration from “Framework Layer” to “Hardware Layer”

The Qualcomm AI Stack adopts a “layered design,” with each layer performing specific roles and seamless collaboration between layers, forming an ecosystem that is “developer-friendly and fully adapted to hardware.”

The architecture can be divided into four layers from top to bottom: “Framework and Runtime Layer,” “Development Tools and Optimization Layer,” “System Software and Driver Layer,” and “Hardware Adaptation Layer,” with each layer focusing on “maximizing hardware performance and simplifying the development process“:

Layer	Core Components	Function
Framework and Runtime Layer	TensorFlow, PyTorch, ONNX, Keras, TensorFlow Lite, ExecuTorch	Supports mainstream AI frameworks, allowing developers to adapt to Qualcomm hardware without modifying model code
Development Tools and Optimization Layer	Qualcomm Neural Processing SDK, AIMET, Qualcomm AI Engine Direct SDK	Provides model optimization (quantization, compression), debugging, and deployment tools to enhance model performance and energy efficiency
System Software and Driver Layer	RTOS, device drivers, LLVM compiler, OpenCL, DirectML	Connects software and hardware, optimizing scheduling for CPU/GPU/NPU, supporting cross-platform hardware access
Hardware Adaptation Layer	Dedicated adaptation modules for Hexagon NPU, Adreno GPU, Oryon CPU	Ensures software fully utilizes hardware features (such as INT4 acceleration, tensor operations), avoiding performance loss

2.2 Upper Layer: Framework and Runtime Support—Lowering Development Barriers

The core of the upper layer of Qualcomm AI Stack is “compatibility with mainstream frameworks,” allowing developers to deploy existing AI models on Qualcomm platforms without learning new tools. The design thinking of this layer is to “minimize developers’ migration costs”:

Upper Layer Thinking 1: Supporting Mainstream Frameworks and Runtimes

Qualcomm collaborates deeply with framework vendors such as Google, Meta, and Microsoft to provide “native support” for mainstream frameworks like TensorFlow, PyTorch, ONNX—developers only need to use a simple “delegator” to delegate the model’s computation tasks to the Qualcomm AI Engine (rather than CPU/GPU), fully leveraging NPU performance. For example, a model developed using TensorFlow Lite only needs to add one line of code (<span>interpreter.set_delegate(QualcommAIDelegate())</span>) to allow the NPU to perform inference without modifying the model structure.

At the same time, Qualcomm supports emerging runtimes (such as ExecuTorch, ONNX Runtime) to adapt to the deployment needs of generative AI. For example, ExecuTorch is a mobile PyTorch runtime launched by Meta, and Qualcomm optimized the NPU delegator of ExecuTorch to improve the inference speed of Llama 2-7B on Snapdragon 8 Gen 3 by 20%.

Upper Layer Thinking 2: Cross-Platform Consistency

Whether developers are developing for smartphones (Android), PCs (Windows), or cars (QNX), the framework support layer of Qualcomm AI Stack provides a consistent interface— for example, a model deployed using TensorFlow Lite on Android can run on the Snapdragon X Elite platform on Windows PC with slight adjustments to driver configurations, without needing to retrain the model. This “cross-platform consistency” significantly reduces the development costs of multi-device AI applications.

2.3 Middle Layer: Development Tools and Optimization Layer—Releasing Hardware Performance

If the framework layer is “lowering the threshold,” then the development tools and optimization layer is “releasing performance”—through model optimization and debugging tools, allowing developers to fully utilize the hardware features of the Hexagon NPU (such as INT4, tensor acceleration) without sacrificing accuracy. The core tools of this layer include:

Tool 1: Qualcomm Neural Processing SDK: Core for Deployment and Debugging

This is Qualcomm’s core SDK for AI development, supporting Android, Linux, and Windows systems, providing three main functions:

1. Model Conversion: Converts TensorFlow and PyTorch models into Qualcomm’s proprietary “DLVM” format, optimizing NPU execution efficiency;
2. Inference Deployment: Provides C++/Java API, supporting synchronous/asynchronous inference of models, adapting to real-time application scenarios (e.g., AI assistants);
3. Performance Analysis: Built-in “AI Profiler” tool allows real-time monitoring of NPU, GPU, and CPU’s computational load and memory usage, helping developers identify performance bottlenecks—for example, discovering high memory read latency for LLMs can be optimized through INT4 quantization.

Tool 2: AIMET: The “Weapon” for Model Optimization

AIMET (AI Model Efficiency Toolkit) is an open-source model optimization tool launched by Qualcomm, with the core function of “reducing model computational and memory requirements without sacrificing accuracy,” designed to enable large models to run on-device. AIMET supports four core optimization techniques, all targeting generative AI model optimization:

• Quantization: Supports INT4/INT8/FP16 quantization, reducing precision loss through “quantization-aware training (QAT).” For example, quantizing Llama 2-7B from FP32 to INT4 reduces memory usage from 28GB to 3.5GB, with precision loss < 2%, while inference speed increases by 3 times;
• Compression: Removes redundant weights from the model through pruning, for example, compressing Stable Diffusion’s parameters by 50% while maintaining image generation quality;
• Neural Architecture Search (NAS): Automatically designs model architectures suitable for Qualcomm hardware, for example, generating a “lightweight LLM” for mobile, reducing parameter scale from 7B to 1B, doubling inference speed while maintaining conversational coherence;
• Conditional Compute: Dynamically activates model layers based on input content, for example, in the AI assistant scenario, activating only 10% of the model layers for simple questions and 100% for complex questions, reducing average energy consumption by 40%.

Tool 3: Qualcomm AI Engine Direct SDK: Deep Hardware Control

For developers requiring extreme performance (such as OEM manufacturers), Qualcomm provides the AI Engine Direct SDK, allowing direct control over the computational allocation and memory scheduling of NPU, GPU, and CPU. For example, developers can specify that the “tensor operations” of LLM run on the NPU, the “attention mechanism” runs on the GPU, and the “text preprocessing” runs on the CPU, achieving fine-tuned scheduling of heterogeneous computing power to further enhance performance.

2.4 Lower Layer: System Software and Driver Layer—The “Link” Connecting Hardware and Software

The lower layer of Qualcomm AI Stack is the “system software and drivers,” responsible for translating the development needs of the upper layer into hardware instructions, ensuring that software fully utilizes hardware features. The core components of this layer include:

Component 1: Optimized Compilers and Drivers

Qualcomm has developed dedicated compilers for Hexagon NPU, Adreno GPU, and Oryon CPU based on the LLVM compiler framework, optimizing instruction generation for AI workloads. For example, for the NPU’s tensor accelerator, the compiler automatically converts matrix multiplication operations into hardware-supported instructions, avoiding efficiency losses from software simulation; for AI tasks on the CPU, the compiler optimizes loop unrolling and vector instructions to enhance the inference speed of small models.

At the same time, Qualcomm provides “low-latency drivers” for each processor, for example, the scheduling latency of the NPU driver is < 1ms, supporting real-time AI applications (such as real-time translation).

Component 2: Cross-Platform OS Support

Qualcomm AI Stack supports four major OS: Android, Windows, Linux, and QNX, covering device types such as smartphones, PCs, cars, and IoT. For example, on Windows PCs, the AI capabilities of Adreno GPU can be called through the DirectML interface; in cars, the NPU can run autonomous driving perception models through QNX’s real-time drivers. This cross-OS support allows Qualcomm AI technology to cover all scene devices.

Component 3: Memory and Computational Scheduling

The system software layer includes a built-in “heterogeneous computing scheduler” that dynamically allocates computational and memory resources for NPU, GPU, and CPU based on application needs. For example, when running an AI assistant, the scheduler prioritizes allocating computational power to the NPU; when running games, it allocates AI computational power (such as super-resolution) to the GPU; when running text completion, it allocates computational power to the CPU. This dynamic scheduling ensures that “computational power is used where it is needed,” avoiding resource waste.

2.5 Core Value: “One Development, Multi-End Deployment” and Scalable Empowerment

The ultimate goal of Qualcomm AI Stack is to “lower the deployment threshold for on-device AI and promote large-scale implementation.” By 2024, Qualcomm has opened the AI Stack to developers worldwide, supporting over 2 billion Qualcomm platform devices (smartphones, PCs, cars, XR devices), forming a vast on-device AI ecosystem. Its core value is reflected in two aspects:

• Lowering development costs: Developers do not need to develop different versions of AI models for different devices (phones, PCs, cars); they only need to complete one development and optimization through AI Stack to deploy across all Qualcomm platform devices. For example, an AI image generation application developed on Snapdragon 8 Gen 3 can run on Snapdragon X Elite PC without code modification, automatically adapting to the PC’s higher computational power (increasing image generation speed from 0.6 seconds to 0.3 seconds).
• Releasing hardware performance: Through the optimization tools of AI Stack (such as AIMET), developers can fully utilize the features of Qualcomm hardware, avoiding “hardware performance waste.” For example, the unoptimized Llama 2-7B has an inference speed of 10 tokens/s on Snapdragon 8 Gen 3, while after optimization with AIMET’s INT4 quantization and microtile inference, the speed increases to 20 tokens/s, with precision loss < 2%.

unsetunset3. Performance: Comprehensive Leadership from Benchmark Testing to Actual Applicationsunsetunset

The ultimate standard for measuring AI technology is “actual performance”—including both theoretical performance in benchmark tests and user-perceptible actual application experiences. Qualcomm has achieved comprehensive leadership in AI performance in both smartphones and PCs through the collaboration of Hexagon NPU and AI software stack, demonstrating advantages of “strong computing power, high energy efficiency, and good experience” in both benchmark tests and generative AI applications.

3.1 Mobile End: AI Performance Breakthrough of Snapdragon 8 Gen 3

The Snapdragon 8 Gen 3 is the benchmark for current mobile AI performance, with its Hexagon NPU significantly outperforming competitors in both benchmark tests and actual applications, primarily due to the deep collaboration of “hardware optimization + software adaptation.”

(1) Benchmark Testing: Leading in Multiple Dimensions

In mainstream AI benchmark tests, the Snapdragon 8 Gen 3’s performance far exceeds that of competitors, especially in tests related to generative AI:

• MLPerf Inference Mobile V3.1 (Authoritative AI Benchmark): MLPerf is an authoritative benchmark launched by the MLCommons alliance, covering tasks such as language understanding and image classification. In the key model MobileBert (language understanding) test, Snapdragon 8 Gen 3’s performance is 17% higher than competitor A and 321% higher than competitor B; in the image classification model ResNet-50 test, performance is 280% higher than competitor B.
• Ludashi AIMark V4.3 (Comprehensive Mobile AI Test): This test covers AI computing, image processing, voice recognition, etc., with Snapdragon 8 Gen 3’s overall score being 5.7 times that of competitor B and 7.9 times that of competitor C.
• AnTuTu AITuTu (Mobile AI Specialized Test): Snapdragon 8 Gen 3’s score is 6.3 times that of competitor B, with specialized scores for generative AI (such as text generation, image generation) being 8.1 times that of competitor B.

These benchmark results indicate that Snapdragon 8 Gen 3’s AI computing power is not only “high at peak” but also “broad in coverage of scenarios,” especially showing significant advantages in generative AI tasks.

(2) Actual Generative AI Applications: User-Perceptible Smooth Experience

Benchmark tests are theoretical, while actual applications are the core of user experience. The Hexagon NPU of Snapdragon 8 Gen 3 performs excellently in mainstream generative AI applications, achieving “on-device real-time generation”:

• LLM Voice Assistant: Running the Llama 2-7B model, the token generation speed reaches 20 tokens/s, enabling “responses generated within 1 second after voice input,” far exceeding competitors (competitor B at about 8 tokens/s, requiring 2-3 seconds for a response); at the same time, the power consumption for continuous operation for 1 hour is only 5% of battery power, supporting long conversations.
• Image Generation (Stable Diffusion): Generating a 512×512 resolution image takes only 0.6 seconds, over 4 times faster than competitor B (about 2.5 seconds), with a CLIP accuracy of 0.89 (close to the desktop model’s 0.92), producing clear image details without significant distortion.
• Real-Time Translation: Running a multimodal translation model (supporting voice-text-voice), achieving real-time translation for 20 languages with a delay of < 300ms, over 2 times faster than competitor B (delay > 800ms), with an accuracy rate of 95% (close to professional translation level).
• Video Super Resolution: Running an AI video super-resolution model, capable of real-time upgrading 720P video to 4K, maintaining a frame rate of 60fps, with power consumption 35% lower than competitor B, avoiding phone overheating.

3.2 PC End: AI Performance Revolution of Snapdragon X Elite

The Snapdragon X Elite is Qualcomm’s first “AI-first” platform for PCs, with its Hexagon NPU and Oryon CPU collaborating to break the X86 PC’s monopoly on AI performance, especially demonstrating a dual advantage of “high performance + low power consumption” in generative AI tasks.

Benchmark Testing: Crushing X86 Competitors

In PC AI benchmark tests, the Snapdragon X Elite’s performance far exceeds that of mainstream X86 competitors:

• UL Procyon AI Inference Benchmark (Windows PC AI Benchmark): This test covers LLM inference, image recognition, voice processing, etc., with Snapdragon X Elite’s overall score being 3.4 times that of X86 competitor A and 8.6 times that of X86 competitor B. Among them, the LLM inference score is 4.2 times that of X86 competitor A, and the image generation score is 9.1 times that of X86 competitor B.
• TOPS Computing Power Comparison: The Hexagon NPU of Snapdragon X Elite achieves 45 TOPS (INT8), far exceeding X86 competitor A’s 28 TOPS and competitor B’s 15 TOPS; if using INT4 precision, computing power can be increased to 90 TOPS, further widening the gap.

Actual Generative AI Applications: The “On-Device AI Revolution” on PC

Snapdragon X Elite achieves a breakthrough in generative AI applications on PC, realizing “desktop-level experience + mobile-level power consumption”:

• LLM Office Assistant: In collaboration with Oryon CPU and Hexagon NPU, running the Llama 2-7B model, the token generation speed reaches 30 tokens/s, 2.5 times faster than X86 competitor A (about 12 tokens/s); running the 13B parameter Llama 2-13B model, the token generation speed reaches 18 tokens/s, supporting more complex office scenarios (such as document summarization, code generation), while X86 competitor A experiences stuttering (about 5 tokens/s) when running Llama 2-13B.
• Image Generation (Stable Diffusion): Generating a 512×512 image takes only 0.9 seconds, while generating a 1024×1024 image takes only 3.2 seconds, over 3 times faster than X86 competitor A (512×512 takes 2.8 seconds, 1024×1024 takes 8.5 seconds); at the same time, generating 10 images of 1024×1024 consumes only 8Wh, 64% lower than X86 competitor A (22Wh), avoiding PC overheating.
• AI Code Generation: Running the CodeLlama 7B model, capable of real-time generation of Python and Java code, with response latency < 500ms, achieving an accuracy rate of 85% (close to professional developer level), making it more suitable for programming scenarios than X86 competitor B (latency > 1.2 seconds, accuracy rate 72%).
• Multimodal Meeting Assistant: Simultaneously running “speech-to-text (Whisper),” “LLM summarization,” and “real-time translation” models, with power consumption only 15W, 57% lower than X86 competitor A (35W), supporting 4 hours of continuous meetings, while X86 competitor A can only support 1.5 hours.

3.3 Practical Cases of Heterogeneous Computing: Synergistic Effects of Avatar AI Assistant

Qualcomm showcased the “Avatar AI Assistant” at the Snapdragon Summit 2023, a typical case of heterogeneous computing—achieving a smooth experience of “voice interaction + real-time virtual avatar rendering” through the collaboration of Hexagon NPU, Adreno GPU, Oryon CPU, and Sensing Hub, fully demonstrating the comprehensive advantages of Qualcomm’s AI technology system.

Task Allocation: Each Plays Its Role, Maximizing Efficiency

The core tasks of the Avatar AI assistant include “voice input → speech-to-text (ASR) → LLM generates response → text-to-speech (TTS) → virtual avatar rendering,” with Qualcomm allocating these tasks to different processors to maximize efficiency:

1. Sensing Hub: Responsible for real-time listening to voice commands, running the Whisper ASR model (lightweight version), with power consumption < 1mA, ensuring “always-on” without consuming power;
2. Hexagon NPU: Responsible for two core tasks—running the Llama 2-7B model to generate text responses (token speed 20 tokens/s) and running the “audio-to-blendshape” model (converting speech rhythm into virtual avatar’s lip and expression data), ensuring synchronization of responses and expressions;
3. Oryon CPU: Responsible for running the open-source TTS model, converting text responses into natural speech, with latency < 200ms, avoiding desynchronization of speech and avatar;
4. Adreno GPU: Responsible for rendering the virtual avatar (based on Unreal Engine MetaHuman), maintaining a frame rate of 60fps to ensure smooth and natural appearance;
5. Memory Subsystem: Achieving real-time data transfer between processors through high-bandwidth shared memory, avoiding data latency.

Experience Effect: Real-Time, Smooth, Low Power Consumption

Through heterogeneous collaboration, the Avatar AI assistant achieves a total latency of < 1.5 seconds for “user speaking → virtual avatar responding,” with power consumption for continuous operation for 1 hour being only 6% of battery power (on smartphone) or 12Wh (on PC). In comparison:

• If only the NPU runs all tasks, latency would increase to 3.5 seconds, and power consumption would increase by 80%;
• If only the CPU runs, latency would increase to 5 seconds, and power consumption would increase by 200%.

This case fully demonstrates that heterogeneous computing is the “optimal solution” for on-device generative AI, and Qualcomm’s AI technology system (NPU + software stack + heterogeneous architecture) is the core support for achieving this solution.

unsetunset4. Conclusion: Core Competitiveness and Future Direction of Qualcomm’s AI Technology Systemunsetunset

Qualcomm has built a complete technology system for on-device generative AI through “Hexagon NPU innovation,” “Qualcomm AI software stack empowerment,” and “heterogeneous computing collaboration” as three pillars. Its core competitiveness lies not in a single hardware or software, but in the system-level integration capability of “from application to hardware”—continuously tracking AI model trends, dynamically adjusting the design focus of the NPU; through full-stack software tools, lowering the development threshold for developers; and maximizing computational efficiency through heterogeneous collaboration.

This approach of “hardware following model trends, software following hardware trends, and ecosystem following developer trends” keeps Qualcomm at the forefront of on-device AI.

Looking ahead, Qualcomm’s AI technology will deepen in three directions:

1. Support for Larger Models: Through NPU architecture upgrades and memory optimizations, support for LLM/LVM with over 10 billion parameters to run on-device, further narrowing the AI capability gap between on-device and cloud;
2. Multimodal Collaborative Optimization: Optimizing task allocation strategies for cross-modal tasks involving text, voice, images, and video, achieving “multimodal input → multimodal output” for real-time processing on-device;
3. Full-Scene Ecosystem Expansion: Extending AI technology from smartphones and PCs to cars, XR, robotics, etc., through a unified AI software stack, achieving “AI collaboration across all scene devices” (e.g., seamless connection between mobile AI assistants and car AI assistants).

In the wave of generative AI penetrating on-device, Qualcomm’s AI technology system not only provides core computational support for device manufacturers but also offers developers tools for scalable deployment, ultimately transforming “on-device generative AI” from concept to a user-perceptible smooth experience—this is the core vision of Qualcomm’s “making intelligent computing ubiquitous” and the key to its continued leadership in the AI era.

To join the discussion group, please reply in the NeuralTalk public account background: Let’s be friends