NPUEval: Evaluating AMD Hardware LLM Vectorization NPU Kernel

Keywords: NPUEval, NPU Kernels, LLMs, Vectorization, Open Source Compilers, Performance Evaluation

[NeuralTalk Brief Review]The AMD NPU’s AIE tile relies on VPU (Vector Processing Unit), and vectorizing code is key to unleashing its computing power while reducing latency and power consumption. However, the NPU programming ecosystem is immature and fragmented, leading to LLM-generated code that is often “functionally correct but inefficient”. Only vectorized kernels can match the parallel architecture designed for AI workloads, which is why this paper focuses on vectorization in code generation.

• NPUEval: Optimizing NPU Kernels with LLMs and Open Source Compilers
• https://arxiv.org/abs/2507.14403
• https://amdresearch.github.io/NPUEval/
• https://github.com/AMDResearch/NPUEval
• This article has 9470 words and requires 32 minutes to read, Podcast 7 minutes

Related Articles

• End-to-end acceleration of 4.5 times! Edge SoC dynamic sparse attention shadowAttn: Key token sparse computation architecture compatible with mainstream frameworks unleashes NPU performance limits!
• Qualcomm Edge Generative AI: Hexagon NPU architecture innovation and heterogeneous implementation achieve 98% performance improvement, 0.6 seconds image generation, and 20 tokens/s LLM inference
• HotChips 2025: From PyTorch to CUDA: The technological leap of automatic generation of GPU kernels in the era of large models, Tri Dao’s latest speech

Neural Processing Units (NPUs) are becoming increasingly important in power-sensitive devices such as consumer electronics, and the definition of AI PCs includes such dedicated processors. To efficiently run AI workloads on these devices, a library composed of optimized kernel functions is required.

Creating efficient kernel functions not only requires developers to master domain-specific C++ languages that include vector intrinsics but also to have a deep understanding of the target architecture.

Unlike GPU programming, which has matured over years of development, NPU programming is still in its nascent stage, with smaller and more fragmented developer communities across different hardware platforms.This fragmentation poses challenges for utilizing large language models (LLMs) to assist in writing NPU kernel functions—because domain-specific optimized code examplesare extremely rare in LLM pre-training data.

This paper proposes NPUEval—a benchmark suite for writing and evaluating NPU kernel functions that includes 102 operators commonly used in machine learning workloads. We utilize an open-source compiler tool aimed at AMD NPU to evaluate LLM-generated code from the perspectives of functional correctness and vectorization efficiency.

We evaluate a range of state-of-the-art LLMs, covering proprietary models and open-source weight models. The latest inference models like DeepSeek R1 have shown promising results, achieving over 50% vectorization rate out-of-the-box (out-of-the-box) on some kernel functions.However, even with compiler feedback and vectorized kernel function examples, the average vectorization score across the entire dataset remains around 10%—indicating that even for cutting-edge models, this dataset remains a daunting challenge.

The dataset and evaluation code will be released under a permissive open-source license, providing a critical benchmark for advancing research in code generation and NPU kernel function optimization.

unsetunsetTable of Contentsunsetunset

• Table of Contents
• 1. Introduction
• 2. Vectorization
• 3. Dataset
• 3.1 Dataset Structure
• 3.2 Dataset Considerations
• 4. Kernel Evaluation
• 4.1 Evaluation Toolchain Setup
• 5. Generation
• 5.1 System Prompts
• 5.2 Retrieval-Augmented Generation
• 5.3 Compiler Feedback
• 5.4 LLM Settings
• 6. Results
• 6.1 Out-of-the-Box LLM Evaluation
• 6.2 Functional Correctness
• 6.3 Performance
• 6.4 Analysis of Generation Failures
• 7. Limitations
• 8. Future Work
• 9. Conclusion
• References

To join the discussion group, please reply in the NeuralTalk public account backend: Make a Friend

unsetunset1. Introductionunsetunset

Large language models (LLMs) have become highly attractive code generation assistants, achieving breakthrough performance in various coding benchmarks [1][2][3][4][5][6][7][8]. For more challenging benchmarks like SWE-Bench [9], which focus on problem-solving, the success rate of cutting-edge models has exceeded 62% and is still rapidly improving [10]. However, despite their excellent performance on typical Python tasks, many benchmarks prioritize “pass/fail” metrics over code quality.A poorly written algorithm may pass functional tests but is unsuitable for production systems. This is particularly tricky when generating efficient kernel functions for hardware accelerators—unoptimized kernel functions are useless in acceleration applications.

Currently, major chip manufacturers [11][12][13][14][15][16] are launching NPUs to support AI acceleration workloads. However, existing research on optimizing kernel function generation primarily targets GPUs [17][18][19].

• GPU programming languages and their ecosystems have matured over years of development, with high coverage in LLM pre-training data.
• In contrast, the developer community for NPU programming is smaller and more fragmented due to hardware architecture differences, and the software stack is not mature enough.

We find that even though cutting-edge models can solve general software problems well, they often struggle to generate optimized solutions for vendor-specific code.

This paper proposes NPUEval—a dataset designed specifically to evaluate the ability of LLMs to generate AMD NPU vectorized kernel functions. Due to the accessibility of NPUs in consumer devices, these accelerators become an ideal platform for kernel code generation research—ensuring reproducibility of research by running on consumer laptops and mini PCs. The entire evaluation toolset (including compilers) is built on open-source tools, and we will release the dataset and related code under a permissive license. The dataset includes prompts for each kernel, behavioral models, and data movement information.

Our evaluation framework measures LLM performance from two dimensions:

• First is functional correctness (similar to benchmarks like HumanEval [1])
• Second is cycle-accurate performance metrics (such as vectorization score—determined by the percentage of cycles spent executing vector instructions out of total cycles)

Our research results indicate that LLMs can generate functionally correct code that runs on NPU hardware, and by adding simple techniques like compiler feedback and retrieval-augmented generation (RAG),they can write vectorized implementations.

The contributions of this paper are as follows:

• Propose NPUEval—this is the first benchmark suite for evaluating the ability of LLMs to generate NPU kernel function vectorization code.
• Provide a fully open-source technology stack for AMD NPU programming (compilers and drivers) focused on single kernel function development, which can run on laptops.
• Develop a comprehensive evaluation toolset, supporting correctness evaluation, cycle-accurate performance metric calculation, and kernel microcode output, which can be further optimized based on agentic workflows.
• Design a reference LLM pipeline,which utilizes compiler feedback and retrieval-augmented generation (RAG) of vectorized kernel function examples to guide LLM outputs towards vectorization schemes and reduce hallucinations in generated content.

unsetunset2. Vectorizationunsetunset

The hardware targeted in this paper is the latest AMD laptops and mini PCs (such as devices based on the Phoenix, Hawk Point, or Strix Point architectures) that internally use AI Engines (AIE) to perform kernel computations.

For example, devices based on the Phoenix architecture contain 20 AIE tiles—each tile is a computational unit capable of processing data independently. Each AIE tile is equipped with a Vector Processing Unit (VPU) and a scalar unit, which provides high flexibility as the scalar unit can execute any C++ code.

Similar to GPUs, these AIE tiles also require kernels; and to fully utilize the hardware architecture for maximum performance, it is crucial to write these kernels using optimized vectorized code. Inefficient kernels significantly increase latency and power consumption, which completely contradicts the purpose of integrating AIE into devices. However, to extract performance as much as possible, we need to maximize the use of the VPU for computation,which means writing vectorized code using the C++ AIE API and vector intrinsics.

Figure 1 shows a simple example of vectorization.

• In which, the scalar code (Figure 1a) is a passthrough kernel that iterates element by element through a for loop, copying input data to the output buffer;
• While the vectorized code in Figure 1b achieves the same kernel functionality but utilizes the VPU through vector instructions—this code processes a block of data at once rather than a single element, significantly improving throughput.

Optimized kernels for vectorization may be more complex than this simple example, with more lines of code, but the vectorization method remains consistent (refer to the vectorization code example below). The challenge faced by large language models (LLMs) is that the generated vectorized code must meet both functional correctness and efficient utilization of the VPU while using the correct API.

unsetunset3. Datasetunsetunset

The NPUEval dataset is a collection of prompts that include:

• AIE kernel definitions, accompanied by documentation strings (docstrings);
• Documentation strings that include kernel descriptions, input/output examples, and expected data movement and runtime parameters.

The following figure shows an example of a prompt.

3.1 Dataset Structure

Each kernel in the dataset contains the following components:

• Prompt: Inspired by HumanEval [1], the prompt contains a basic description of the kernel functionality and function signature.
• Data movement information: Specifies the data size entering and exiting the AIE tiles.
• Behavioral model: A Python implementation of the target kernel based on NumPy.
• (Optional)Canonical C++ solution: The Python solution is used for correctness evaluation, while the canonical C++ solution is used for regression testing of the harness.

To generate optimized kernels that fully utilize the target hardware, kernels may need to be written for specific fixed input data sizes. Therefore, we include data movement information for each kernel to specify the data sizes entering and exiting the tiles [20].

3.2 Dataset Considerations

Since the core task is to write code for dedicated hardware, several issues and challenges that are typically not addressed in conventional LLM code generation benchmarks must be considered when constructing this dataset.

3.2.1 Data Type Support

In addition to common NPU data types like int8, which are often used for quantizing neural networks, AMD’s NPU also supports bfloat16, a relatively new data type.

Currently, Python libraries like NumPy do not natively support these new machine learning data types. To generate behavioral models for bfloat16, we used Google Jax’s ml_dtypes library [21]—which provides a NumPy-compatible bfloat16 type.

Additionally, AMD NPU’s bfloat16 implementation supports programmable rounding modes,and we leverage this feature to ensure that the same operation produces completely consistent outputs, minimizing accumulated rounding errors.

3.2.2 Floating Point Kernel Evaluation

The output results of floating-point kernels may not match exactly with the outputs of Python behavioral models. The same problem can be solved by different algorithms, which may have slight differences in precision.

• For example, when calculating trigonometric functions like sin, the NPU may use a different polynomial approximation method than the CPU implementation in the behavioral model;
• Moreover, floating-point operations do not satisfy the associative law, so different operation orders can also lead to slight differences in results.

When evaluating the correctness of generated kernels:

• We adopted a default absolute error threshold of 1e-2 (consistent with the default threshold in related kernel generation research [17]).
• This default threshold applies to logical operators and integer operators, but for low-precision floating-point operations, we set the tolerance to 2e-2 to provide some flexibility for LLMs in the algorithm search space;
• For more complex activation function kernels like tanh or softmax, the tolerance is set to 3e-2.

3.2.3 Data Movement

To compile AIE graphs using the MLIR-AIE tool [22], we predefined data movement rules for each kernel and included them in the prompts.

• Test vectors are used to infer data movement and automatically generate the underlying MLIR code for configuring the device.
• In dataflow programming, kernels need to know the data sizes entering and exiting the tiles [23]—this is crucial for compiler optimization of buffer allocation and bank assignment [20].

3.2.4 Hardware Availability

Most code generation benchmarks for Python or other high-level language problems are convenient because they can run on any commercial hardware. However, writing code for hardware accelerators poses challenges:it requires testing on the target device or using simulators (which are often much slower and limit iteration counts).

Although AIE can also be found on embedded application development boards [24], these platforms are less accessible to mainstream users.

We chose AMD NPU as the target hardware because it is easily accessible on consumer-grade hardware, allowing a broader range of researchers to reproduce our experimental results. The results presented in this paper were obtained using a laptop equipped with a Ryzen 9 7940HS chip.

unsetunset4. Kernel Evaluationunsetunset

The generated kernels will be evaluated based on the following criteria:

• Compilation—Is the kernel compilable? Is it syntactically correct C++ code using valid NPU vector unit API calls and intrinsics?
• Functional correctness—Can the kernel generate correct outputs for given inputs within built-in error tolerances?
• Performance—How long does it take to execute the kernel? What is the utilization of the Vector Processing Unit (VPU)?

4.1 Evaluation Toolchain Setup

The high-level overview of the evaluation pipeline is shown in Figure 3.

• The dataset includes kernel prompts, behavioral models, and test vectors.
• The evaluation toolchain compiles the generated C++ kernel code and runs it on the target hardware (on-target).
• Subsequently, the output results are compared with the expected simulated outputs generated by the Python behavioral model.

Category	Subitem	Details
Dataset	Prompt	Inspired by HumanEval, it contains kernel functionality descriptions and function signatures, providing the foundational input context for LLM to generate NPU kernel code
Behavioral	Based on NumPy (or Google Jax ml_dtypes library, supporting bfloat16) Python implementation, serving as a reference benchmark for NPU kernel outputs for correctness validation
Reference inputs	Test vector inputs included in the dataset, used to drive NPU kernel execution, consistent with the inputs of the behavioral model
Reference output	Expected outputs generated by the behavioral model based on reference inputs, used to compare with the actual outputs of the NPU kernel to determine functional correctness
Evaluation System	LLM	The code generation entity that receives information such as prompts from the dataset and outputs the target NPU kernel’s C++ code
C++ Code	The core output product of the LLM, which must comply with AIE vector API and intrinsic specifications, serving as the basis for subsequent compilation and execution
LLVM-AIE Compiler	Used to compile the C++ code generated by the LLM, a branch of LLVM specifically adapted for AIE kernel programming, fully open-source (installable from GitHub repository)
MLIR-AIE	Open-source framework for configuring AIE arrays and loading kernels into the correct AIE tiles, the core configuration tool for NPU programming
Kernel Object	Intermediate product generated after compiling C++ code with the LLVM-AIE compiler, not yet a file that can be directly run on the NPU
Binary	Final executable file generated after further processing of the kernel object file, which can be directly loaded and executed on the NPU
NPU	The target hardware where the code runs (e.g., NPU of AMD Phoenix, Hawk Point architecture devices), responsible for executing the binary file and outputting computation results
Performance	Cycle Count	Cycle statistics of the NPU kernel execution process, a fundamental metric for measuring kernel running efficiency
Total Cycles	Total cycles consumed from the start to the completion of NPU kernel execution, reflecting the overall running time of the kernel
VPU Cycles	Cycles consumed by the Vector Processing Unit (VPU) executing instructions, the ratio of VPU cycles to total cycles is the key metric for evaluating kernel vectorization efficiency
Microcode	Low-level instructions during NPU kernel execution, useful for analyzing kernel instruction-level execution details to assist in performance optimization
Compiler Feedback	If the C++ code generated by the LLM fails to compile, the LLVM-AIE compiler will output error messages, providing feedback to the LLM for iterative code correction (up to 10 retries, with diminishing returns after 10 attempts)
Correctness	Output Data	The actual computation results output by the NPU after executing the kernel, which need to be compared with the reference outputs of the behavioral model
Pass/Fail	The determination result after comparing output data with reference outputs: if the error is within the preset threshold (e.g., int8 is 1e-2, bfloat16 is 2e-2), it is “pass”; otherwise, it is “fail”
Error Stats	Includes maximum absolute error (the numerical difference between output values and reference values) and maximum relative error (percentage difference relative to reference values), used to quantify the degree of deviation between output data and reference outputs
Additional Metrics	No additional details yet, can be supplemented based on actual evaluation needs (e.g., error distribution, outlier ratio, etc.)
Compiler	LLVM-AIE Compiler	The same LLVM-AIE compiler mentioned in the “Evaluation System”, can reproduce the code generation steps with compiler feedback on x86 machines, even without an NPU, to verify code syntax correctness
Application Builder	MLIR-AIE Framework	Same as the MLIR-AIE in the “Evaluation System”, used in conjunction with IRON to complete AIE array configuration
IRON Bindings	Tools that assist MLIR-AIE in AIE array configuration and kernel loading, enhancing the convenience and compatibility of the configuration process
Templated Graphs	Provided by the NPUEval evaluation toolchain, supporting most kernel data types and input/output port counts, simplifying the configuration process of MLIR-AIE
Data Movement Information	Generated by parsing the test vectors and behavioral model outputs in the dataset, passed to MLIR-AIE to optimize buffer allocation and storage bank assignment, ensuring efficient data movement in and out of AIE tiles
Runtime	Python bindings for the NPU driver	Implementing data input/output and kernel execution control for the NPU, providing an interface for the Python environment to reduce operational complexity
NumPy Array Handling	With Python bindings, easily handle NumPy formatted input/output data, adapting to the Python ecosystem of behavioral models
Output Comparison Validation	Quickly compare NPU output data with reference outputs from the behavioral model, automatically completing correctness determination (Pass/Fail)
Performance Metrics	Execution Time	The actual time taken from kernel startup to output results, an efficiency metric that users can intuitively perceive, corresponding to cycle counts (needs to be converted based on NPU clock frequency)
Precision Measurement (Maximum Absolute Error, Maximum Relative Error)	Same as the error statistics in “Correctness”, can assist agents in iterative optimization (e.g., if the error is too large, improve the algorithm; if VPU utilization is low, adjust the vectorization strategy)
Post-processing	Code Block Extraction (Regular Expressions)	To address the issue of natural language mixed into LLM outputs, marked with “` code blocks, extract code blocks from Markdown using regular expressions to ensure the code can be used for subsequent compilation
Irrelevant Function Truncation	Truncate additional generated main() and other test functions (LLMs often output main() to test kernels), retaining only the target NPU kernel function to avoid interference with compilation and execution

unsetunset5. Generationunsetunset

In addition to the dataset and evaluation toolchain, we also provide a reference code generation pipeline that can be used for various LLMs.This pipeline includes a system prompt (to guide the LLM to generate standalone C++ solutions), a vector database built on open-source AIE kernels (for retrieval-augmented generation, RAG), and compiler feedback provided to the LLM (to reduce errors and hallucinated outputs).

5.1 System Prompts

System prompts are typically used to guide model behavior by providing context and expected outcomes.

Preliminary experiments indicate that adding system prompts is crucial for many LLMs to generate well-formatted code. Some models may attempt to split solutions into multiple code blocks, add explanatory notes, or generate overly verbose outputs, making it difficult to automatically parse solutions. The prompts used in NPUEval are as follows:

1. You are a part of a code generation system for AIE (AI Engines).
2. 
3. * Your job is to write C++ code for a single kernel that will run on an AIE tile.
4. * Produce only the C++ code for the requested kernel including any required headers and imports.
5. * Make sure the C++ code is complete and self-contained in a single code block.
6. * Name the function exactly as specified in the request, and output only the kernel (no main(), examples, explanations or extra code).

Although system prompts cannot guarantee that the model will always follow the rules and generate standalone C++ kernel code blocks, they have proven to be very effective in practice. Some models perform poorly in following instructions, in which case more post-processing is still required (e.g., removing unnecessary <span>main()</span> functions).

5.2 Retrieval-Augmented Generation

The RAG database consists of kernels from open-source GitHub repositories [22][26]. We manually modified these kernels,removing all scalar implementations and retaining only vectorized code examples.

We use llama_index [27] to manage the vector database and example retrieval. The embedding model used is OpenAI’s text-embedding-ada-002 (the default model for llama_index). For each test prompt, two examples are provided to the LLM. The RAG system itself is a rich area for optimization [28], but this is beyond the scope of this paper—we leave it as future work, using the default settings of llama_index during the evaluation process.

5.3 Compiler Feedback

The same compiler (LLM-AIE) is used in both the generation and evaluation phases. The generated kernel code is passed to the compiler, and if compilation fails, error messages are fed back to the model. Each LLM can retry code generation up to 10 times. While more compilation attempts may further improve results,we found that after 10 attempts, the returns begin to diminish.

We further explored the relationship between performance and the number of iterations, as shown in Figure 8, finding a trend of diminishing marginal returns around 5 iterations. Models like DeepSeek V3, Claude Sonnet, and GPT-4.1 are particularly sensitive to compiler feedback, showing significant improvements in their test pass rates.

5.4 LLM Settings

When conditions allow, we lock random seeds to maximize the reproducibility of results.

• For OpenAI models, we set the random seed to 42.
• The temperature parameter for all models is set to 0.0, and top_p is set to 1.0.
• For reasoning models, the temperature parameter is set to the default value of 1.0, and the system prompt is passed as part of the user message.

unsetunset6. Resultsunsetunset

We evaluated mainstream cutting-edge large language models (LLMs) on this dataset, including the latest version of OpenAI’s GPT-4.1, Anthropic’s Claude 3.7 Sonnet, as well as open-source weight models like Meta’s Llama series and other open-source models that excel in code generation tasks (such as DeepSeek R1).

6.1 Out-of-the-Box LLM Evaluation

We first set a benchmark for the models: directly testing the models with the system prompts described in Section 5.1. The benchmark results are summarized in Figure 4.

6.2 Functional Correctness

As shown in Figure 3, the evaluation method for functional correctness is to compare the kernel results output by the programmed NPU with the expected outputs generated by the Python behavioral model.

We explored the effectiveness of compiler feedback through up to five recompilations, with results summarized in Table 1.

Surprisingly, smaller models like GPT-4o Mini and Claude 3.5 Haiku passed more tests in an out-of-the-box scenario than more powerful models like Claude Sonnet and GPT-4o. One reason we observed is that:

• Smaller models tend to generate scalar C++ code by default (i.e., conventional C++ code using for loops rather than vector intrinsics);
• While more powerful models may initially attempt to generate vectorized code, they often end up generating code with “hallucinations” (fictional functions) as shown in Figure 6b.
• After receiving compiler feedback, these stronger models typically revert to scalar solutions, thus passing more tests.

DeepSeek V3, Claude 3.7 Sonnet, and GPT-4o significantly benefited from multiple compilation attempts,while other models showed limited gains. Notably, DeepSeek V3 consistently attempts to include the “adf.h” header file in almost all first attempts—this header file does not exist in the open-source toolset used in the evaluation framework, resulting in a sharp increase in its pass rate to 42% after the first recompilation.

6.3 Performance

The efficiency of generated kernels is calculated by dividing the number of cycles used for VPU execution when processing test inputs by the total number of cycles. Although this method is not absolutely perfect, it effectively reflects the kernel’s utilization of the VPU. Kernels that fail functional tests receive a performance evaluation score of 0%.

The average scores of each model across all kernels are shown in Figure 5. Although these scores may seem low, it is important to note that the vectorization rates of current state-of-the-art (SoTA) open-source kernels [22] typically range from 10% to 30%.

To guide models in generating vectorized code, we introduced a retrieval-augmented generation (RAG) module composed of vectorized kernels into the code generation process. After incorporating the RAG with publicly available AIE kernels into the prompts, all models showed an increase in VPU utilization, with GPT-4.1 showing the most significant improvement in vectorization scores.Strangely, DeepSeek R1 was the only model whose average VPU utilization decreased after introducing vectorization examples—even so, this model still maintains the highest vectorization score without using vectorization examples.

The reason for its performance decline is that some kernels in our database contain compiler-specific compilation directives (pragma), which differ from those used by LLVM-AIE.

After receiving compiler feedback, DeepSeek V3 and GPT-4.1 were among the best-performing models in generating vectorized code, even though their scores in functional correctness tests were relatively low. This phenomenon aligns with the following trend:More powerful models tend to be more “innovative” when attempting to write high-performance code,but due to a lack of deep understanding of the NPU programming environment, they are also more prone to errors.

DeepSeek R1 Result In-Depth Analysis

An unexpected finding in this study is that when provided with vectorized code samples, the utilization of the Vector Processing Unit (VPU) by the DeepSeek R1 model actually decreased. This was the only model among all tested that exhibited a performance decline.

By examining the solutions generated by this model without RAG (Listing 3) and comparing them with those generated with RAG (Listing 4),we found that this model used compiler-specific compilation directives (pragma).Clearly, the R1 model is adept at using compilation directives for low-level optimizations (such as loop unrolling), while the example code in our RAG database is tailored for the Chess compiler—whose set of compilation directives is entirely different from LLVM-AIE (Peano).

Although the Peano compiler does not throw errors when encountering Chess-specific compilation directives, it ignores these directives during optimization. If the large language model is not shown these examples (as shown in Listing 3), it will revert to standard AIE APIs—which have encapsulated optimization operations, eliminating the need to focus on low-level intrinsics or compilation directives.

A potential solution to this issue in future work is to build “compiler backend-specific RAG”.Such information could also be conveyed through system prompts, and as more NPU architectures are released (each with its unique set of intrinsics and compiler-specific optimization paths), this approach is likely to become necessary.

6.4 Analysis of Generation Failures

As shown in Figure 6a,many LLMs that passed functional correctness tests ultimately generated solutions that were extremely inefficient—these kernels failed to fully utilize NPU hardware.Incorrect solutions often resulted from “hallucinations” (as shown in Figure 6b) or misunderstandings of how to use vector APIs (as shown in Figure 6c, where the model successfully used the AIE API but still processed data element by element through loops).

These failures or cases of insufficient utilization of NPU hardware include:

1. Element-wise loops (GPT-4o)—the code example is a scalar loop processing int8 type data;
2. Fictional non-existent APIs (Llama-3.1 70B)—the code example contains fictional API calls;
3. Understanding the concept of vectors and data partitioning but still processing with scalar loops (Claude 3.7 Sonnet)—the code example defines vector sizes but still processes bfloat16 type data element by element).

However, these models seem to possess some inherent knowledge. For instance, Claude 3.7 Sonnet (Figure 6c) can understand the concept of “vector_size” and knows to partition the input buffer into vector blocks of 16 elements.With more advanced prompt engineering and higher-quality examples, these models are expected to generate efficient NPU kernels.

unsetunset7. Limitationsunsetunset

This research primarily targets the AIE architecture in AMD NPUs.Although the methods we designed can be extended to other series of accelerators, and can also adapt when new devices supporting the latest intrinsics and API improvements are released in the AIE series,there are still the following limitations:

• First, the results presented in this paper are based solely on a single compiler backend (LLVM-AIE), and there are multiple compiler choices even within the AIE architecture. As shown by the results of DeepSeek R1, some optimizations in the code are compiler-specific—this will pose an interesting challenge when generalizing the methods to different NPU architectures and programming platforms.

• Second, since NPUs are a relatively new class of devices, their programming models are not as mature as those of other accelerators, and there is currently a lack of high-quality open-source kernels to establish human performance benchmarks. We provided several examples for comparison, including element-wise and convolution operations, but due to length constraints, they are omitted here; it should be noted that:the current state-of-the-art vectorization rates do not exceed 30%, and this rate can vary significantly depending on the type of kernel.

• Finally, our evaluation,while covering a range of mainstream LLMs, still has many more models to be tested. All results are based on greedy decoding of single generations; although “pass@k” evaluations (selecting the best result after multiple generations) may yield higher scores, single decoding still effectively reflects the current capabilities of LLMs in NPU kernel generation tasks.

unsetunset8. Future Workunsetunset

NPUs are still an emerging class of accelerators, and some models have already shown impressive capabilities in writing code for them—this is just the beginning.

In the future, new technologies like code generating agents are expected to further improve the performance of this benchmark, which is an exciting prospect.

Although this research focuses on a single NPU architecture,the method can easily be extended to NPUs from different vendors and series. The Python behavioral model can be reused with the PromptConstructor class to generate datasets for any programming model. We plan to adopt the same approach to extend this work to other series of accelerators.

unsetunset9. Conclusionunsetunset

This paper presents NPUEval—the first systematic benchmark for evaluating the ability of LLMs to generate NPU kernel code. NPUEval includes a comprehensive evaluation framework (providing cycle-accurate performance metrics) and a reference code generation pipeline.

Our results indicate that most LLMs can easily generate scalar code,but struggle to generate optimized vectorized solutions. Interestingly, smaller models like GPT-4o Mini and Claude 3.5 Haiku tend to generate functionally correct but unoptimized code, while more powerful models like GPT-4o, Claude 3.7 Sonnet, and DeepSeek R1 attempt optimization, but score lower in functionality due to hallucinations (fictional content).

We believe that NPUEval provides a valuable foundation for advancing research in LLM-driven accelerator kernel generation. We hope this benchmark will become an industry standard for measuring and improving the code generation capabilities of LLMs for emerging hardware architectures.

unsetunsetReferencesunsetunset

To join the discussion group, please reply in the NeuralTalk public account backend: Make a Friend