1. Current Status and Challenges of AI Infrastructure
Over the past decade, the improvement in AI model performance has relied almost entirely on two main lines: larger data and stronger computing power. However, as the growth of computing power begins to encounter physical bottlenecks while model sizes continue to grow exponentially, the infrastructure for AI faces unprecedented challenges of complexity and fragmentation. From training frameworks, operator libraries, compilers to hardware platforms, the lack of synergy between layers has increasingly revealed issues of high costs and low efficiency, which have become core obstacles to further industry development.
The “Tower Stacking” of Computing Power and Fragmentation of Infrastructure
Today’s AI infrastructure resembles a “tower stacking system”: model → framework → operator library → compiler → hardware. Each layer is iterating rapidly, but the fragmentation of the ecosystem means that each layer of AI infrastructure tools must maintain multiple implementations, leading to an explosion in engineering workload.
First is the diversity of frameworks. Mainstream frameworks include PyTorch, TensorFlow, JAX, etc., which differ in operator granularity, scheduling methods, automatic differentiation mechanisms, and execution models (such as PyTorch’s Eager or Graph mode). For AI infrastructure tools, this means that design cannot be reused, and each set of operator libraries and compilers must be maintained independently, where any change in API or IR may trigger a chain of adaptation costs. In addition to the differences in the frameworks themselves, the diversity of model serialization and exchange formats also brings additional challenges to AI infrastructure. Model files generated by different frameworks (such as PyTorch’s .pt / .pth, TensorFlow’s SavedModel / .pb, JAX’s .msgpack files) differ in their representation of computation graphs, weight storage, and execution signatures, making direct interoperability impossible. To address cross-framework compatibility issues, ONNX, a universal intermediate format, has emerged, attempting to unify the computation graph and weight representation of different frameworks. However, even ONNX may have compatibility differences in dynamic shapes (Shape Infer), operator versions, operator granularity, and extended operations. How to fully retain operator granularity and semantics while ensuring compatibility with different model files and achieve better graph optimization is one of the problems that model compilers need to solve.
Secondly, there is the exponential expansion of operator libraries. Early operator libraries (such as cuDNN, MKL-DNN) mainly focused on basic operators like convolution and GEMM, which were relatively stable. However, since the advent of Transformers, operator combinations have become highly dynamic. For example:
- Attention: Multi-Head → FlashAttention → FlashAttention2 → FlashAttention with Paged KV
- FFN: Standard FFN → SwiGLU → Gated FFN → Blocked FFN
- LayerNorm: LayerNorm → RMSNorm → Pre-RMSNorm → Mixed Normalization Methods
Each structural iteration generates new kernel demands, and these kernels need to be manually optimized for different hardware. This explains why the maintenance cost of operator libraries has increased by more than an order of magnitude over the past decade.
Next is the unsustainability of operator library maintenance costs. In the traditional model, hardware vendors typically need to maintain a separate set of operator libraries for different AI frameworks and inference engines to accommodate different operator definitions and granularities, such as PyTorch, TensorFlow, JAX, onnx, etc. If supporting BF16/FP8/INT4, multiple variants are required. Each new model architecture (such as Mixtral, DeepSeek, LLaMA3, Gemini) may introduce new operators or optimize existing operator implementations, triggering a large-scale update of the operator library.
Finally, the explosive growth of hardware accelerator cards poses challenges. Since 2022, the AI computing power market has entered an “era of a hundred flowers blooming,” with new hardware emerging one after another: GPGPU, ASIC inference cards, FPGA accelerators, RISC-V DSA, DPU, TPU, NPU, etc. Each new architecture means that training frameworks, operator libraries, and compilers must be re-adapted, and this speed even exceeds the iteration speed of Transformer operators, creating industry pain points.
The “Architectural Impact” of Transformers and Large Models
The elegance and flexibility of the Transformer architecture have brought explosive growth in model performance while also placing heavy pressure on infrastructure. Operator combinations are no longer stable but exhibit high dynamism: the FlashAttention series shifts from IO-bound to compute-bound, PagedAttention supports external KV cache, and the structure of attention kernels is constantly changing; dynamic routing brought by MoE (Mixture-of-Experts) further increases operator non-determinism; circular attention, block sparse attention, sliding window attention, grouped Query, decoupled QK scaling, RoPE plugins, and auxiliary weight mechanisms are emerging one after another. Each innovation in papers requires new kernels, and traditional operator libraries struggle to keep up, often resulting in framework-level hacks, performance not meeting standards, or even “stuck” on a certain model version.
More critically, the speed of model changes far exceeds the evolution speed of hardware and compilers. Taking the LLaMA series as an example: LLaMA → LLaMA2 → LLaMA3, FlashAttention from 1 → 2 → 3, LLM, MoE models are constantly evolving, while the design cycle for new hardware typically takes 2.5 to 4 years, and a compiler project takes over 3 years from inception to stability. This speed difference means that operator libraries are always in a state of catch-up.
2. Evolution and Development Challenges of Operator Libraries
As the complexity of models grows exponentially and hardware systems rapidly diversify, the traditional operator library development system can no longer meet the demands of the AI era. This is why, since 2018, AI compiler systems and “operator DSL” systems have emerged one after another, marking the inevitable evolution of operator development from the “craftsmanship era” to the “automation era”.
Bottlenecks of Traditional Operator Libraries: CUDA Monopoly and Limits of Manual Tuning
For over a decade, AI computation has been almost entirely built on CUDA + cuDNN/cuBLAS. NVIDIA’s closed ecosystem has brought extreme performance but has also created industry-level lock-in, limiting operator development efficiency. The optimization of a high-performance CUDA kernel is usually deeply tied to the GPU architecture: SM structure, warp scheduling, bank conflict avoidance, register layout, memory hierarchy, and CUDA Runtime semantics all directly determine the writing of the kernel. In other words, each new hardware, each data type, and even each scheduling strategy means that a new set of operator libraries must be rewritten, leading to development costs that grow almost linearly or even exponentially. The biggest problem with traditional handwritten kernels is not the difficulty of a single kernel, but the exponential explosion in the number of operator combinations. For example, “Matmul + ReLU” is very simple in basic cases, but as model complexity increases, operator combinations can reach thousands or tens of thousands, each requiring a new fused kernel, rewritten for each data type, and retuned for each GPU architecture, ultimately leading to operator libraries that are unsustainable and unscalable, unable to keep pace with the speed of large model development.
Meanwhile, the efficiency of operator development itself is extremely low. A high-performance CUDA kernel often requires manual selection of tile sizes, double buffering, memory access optimization, warp-level reduction and shuffle, and repeated optimization for different batch sizes, shapes, and data types. This artisanal development approach is not only costly but also lacks universality and reusability. More importantly, the speed of innovation is hampered by operator libraries: the speed of algorithm invention far exceeds the speed of operator library updates, and many new models can only use inefficient versions in their early releases, with frameworks potentially falling back to eager mode, leading to a huge gap between performance and model potential. This phenomenon has been particularly evident since the explosion of the Transformer architecture, with the industry widely feeling the pressure of operator library systems “not keeping up”.
The Birth of New Operator DSL: Replacing Manual Optimization with “Descriptive Programming”
To break free from the limitations of handwritten kernels, the industry has begun to explore new operator development methods, leading to the emergence of operator DSL (Domain Specific Language). The design intent of DSL is very clear: developers describe “what to do,” and the compiler decides “how to do it.” It is not a “simpler CUDA” but a more abstract operator description language aimed at completely decoupling operator logic from hardware implementation. Currently, representative DSLs include Triton, TileLang, CuteDSL, and Mojo, as well as NineToothed (九齿), which is a high-level encapsulation based on Triton, and Gluon, which supports specific hardware performance extensions. The common goal of these DSLs is to make operator development simpler and more efficient while providing cross-platform capabilities, hiding most hardware details, and overturning traditional operator development models.
Triton: A Representative of the New Era of Operator DSL
When discussing how operator development models have evolved from traditional engineering processes to higher-level, more abstract expressions, Triton is an unavoidable milestone case. Triton is currently the most successful GPU operator-level DSL, with its core value lying in ease of use, high performance, and significant industry impact. The birth of Triton is not accidental; it is deeply rooted in the industry’s desire for a “higher productivity model development approach.” For over a decade, CUDA has been the de facto standard for high-performance AI kernel development, but it has also deeply bound developers to NVIDIA’s proprietary software and hardware ecosystem. To write high-performance GPU kernels, developers must deeply understand the CUDA programming model, thread scheduling, memory hierarchy, warp behavior, and various hardware details, making operator development a highly specialized and highly demanding task. Meanwhile, these CUDA kernels cannot migrate across hardware, limiting the speed of innovation in the industry and leading the entire AI computing infrastructure towards closure.
It is against this backdrop that OpenAI released Triton—a high-performance DSL (Domain Specific Language) for deep learning operators. The core idea of Triton is very simple yet disruptive: write GPU kernels in Python, allowing the compiler to automatically translate high-level logic into low-level code that fully utilizes hardware performance. It does not merely provide a “sugar-coated” syntax but automates low-level optimizations such as input/output layout, memory access patterns, parallelism strategies, and automatic tiling within the compilation system, freeing developers to shift their focus from “writing thread blocks” and “scheduling instruction order” to “writing algorithms.”
More importantly, Triton demonstrates a key value of DSL in operator development: decoupling algorithm expression from hardware implementation. Traditional CUDA kernels are often tailored for a specific GPU architecture, but Triton allows developers to focus solely on the operator-level description of algorithms, with specific memory layouts, thread models, and instruction scheduling adapted by the compiler for different hardware. This means that once the DSL and compiler ecosystem mature, developers can write an operator once and reuse it across different GPUs, AI accelerators, or even future ISA architectures.
OpenAI has emphasized in its official blog that Triton’s performance can already surpass that of some professionally engineered CUDA kernels, indicating a trend: the performance bottleneck of modern AI operators increasingly comes from compiler optimization rather than manual tuning. This also explains why Triton has become a fundamental component for operator development in various AI frameworks and inference engines—it represents a shift from “hardware binding” to “hardware decoupling” in a new era.
Currently, Triton has been widely applied and has demonstrated its core value: in PyTorch 2.x, Triton has replaced a large number of handwritten CUDA kernels, achieving optimal performance through JIT and AutoTune; in the vLLM inference engine, optimizations for PagedAttention, Softmax, LayerNorm, and KV cache rely on Triton for high-performance computation; in the FlashAttention series, Triton provides a foundation for rapid operator iteration, enabling the research community to complete high-performance prototype development within hours, liberating AI algorithm engineers from the enormous engineering workload of developing new operators, lowering the entry barrier for operator development, and facilitating the development, validation, training, and iteration speed of new model architectures.
3. RISC-V DSA (Domain-Specific Architecture): A New Form of AI Hardware
David Patterson (the father of RISC-V architecture) pointed out in A New Golden Age for Computer Architecture: History, Challenges and Opportunities that this is a golden age for architectures. As the growth of transistor density slows, hardware performance can no longer rely solely on Moore’s Law; in other words, “the free lunch of computing power is over.” Domain-Specific Architectures (DSA) and open ISA/open architecture have ushered in new opportunities.
RISC-V: The True Open Architecture Foundation for the AI Hardware Era
RISC-V is an open-source, reduced instruction set architecture (ISA) that originated in academia and is being developed collaboratively by the global industry. Its core philosophy is “simple, open, and extensible,” which makes it fundamentally different from traditional closed ISAs: it does not rely on expensive licensing and is not limited to specific vendors. The base instruction set remains simple and stable while allowing vendors to freely extend custom instructions as needed, providing great freedom for hardware innovation. In recent years, RISC-V has expanded from embedded devices to high-performance AI accelerators, becoming one of the fastest-growing chip architectures globally.
The potential of RISC-V in the AI acceleration field is particularly prominent, thanks to the emergence of extensions like RISC-V Vector (RVV). RVV supports variable vector lengths, allowing flexible mapping of tensor computations of different sizes at the hardware level. Various vendors are actively experimenting with different matrix extensions that can efficiently accelerate core operators in deep learning (such as GEMM, Attention, convolution, etc.). This scalability means that hardware vendors can freely design AI-specific functions without being constrained by traditional CPU/GPU architectures. Recently, Google’s Coral NPU full-stack open-source project has provided a complete hardware and software implementation of a RISC-V SoC that includes scalar cores, vector execution units, and matrix execution units, offering complete AI acceleration capabilities, further proving RISC-V’s advantages in the AI field. Overall, RISC-V is not just a processor ISA but has become the cornerstone of customizable AI acceleration hardware.
DSA: Exploring New Forms of AI Hardware
Based on the openness of RISC-V, various vendors have begun to attempt to build domain-specific architectures (DSA) for AI. These architectures share common characteristics of being highly customizable and extensible, maximizing performance for specific AI tasks. For example, Tenstorrent’s processors adopt a dataflow architecture, dynamically scheduling operator data flows to achieve high throughput and low-latency matrix computations; mainstream AI accelerator cards generally adopt multi-tile designs, multi-level SRAM hierarchies, and dedicated interconnects to provide elastic resources for large-scale model training and inference. Similar ideas are also appearing in Edge NPUs, cloud ASICs, and FPGA accelerators.
This means that AI hardware is no longer limited to fixed GPU structures but can present a highly heterogeneous and collaborative computing system, with each new hardware potentially becoming a new form of future AI accelerators. Through this approach, RISC-V has become the common language and innovative foundation for the new generation of AI infrastructure, providing unprecedented flexibility and scalability for the entire ecosystem.
4. Compilers: The Key Hub of AI Infrastructure
“Moore’s Law has failed; now is the golden age of compilers.”
——Chris Lattner (father of LLVM, MLIR, Swift)
When hardware performance growth can no longer rely on the increase in transistor density, the software system has become an important means to continuously improve computing efficiency. Over the past decade, hardware innovation has shifted from “Moore acceleration” to “architecture acceleration”; meanwhile, AI models have evolved from convolution stacks to complex operator graph structures, making traditional manual optimization and static library methods unable to keep up with this complexity. This means that the performance bottleneck has shifted from hardware manufacturing to software compilation, and the core path of computing power growth is transitioning from simple hardware optimization to collaborative design and optimization of software and hardware. In the AI era, true competition for computing power is not only a competition of compilers but also a competition of collaborative design and evolution between compilers and hardware architectures.
In the context of operator libraries facing exponential complexity growth and the emergence of new hardware architectures like RISC-V, a previously unprecedented core role has begun to emerge in the AI software stack—the compiler. In the traditional software era, compilers were merely “translators from code to machine instructions”; however, in the AI era, compilers have upgraded to become “intelligent scheduling systems” that span the optimization hub of the entire computing power stack.
The current routes of AI compilers can be roughly divided into two categories: one is model compilers or graph compilers, such as IREE, which focus on optimizations under static structured computation graphs, such as kernel fusion, where the implementation of operators in the model relies on the compiler for algorithm expansion; the other is operator DSL compilers, such as the Triton compiler, which focus on abstract operator language descriptions.In this series of articles, we will focus on operator compilers centered around Triton, and in the future, we will also introduce the development and optimization of model compilers, and even achieve a combination of graph compilers and operator compilers, allowing model compilation to have both graph-level optimization effects and flexible operator implementations.
The importance of compilers does not arise from nowhere but is driven by three factors:
- Operator libraries can no longer bear the explosive adaptation work across models and hardware
- New architectures like RISC-V DSA require a “vendor-neutral” universal software entry
- The complexity of modern hardware has far exceeded the capacity for manual tuning
In this context, compilers have become the only reliable bridge connecting innovative models and rapidly changing hardware.
Connecting Operator DSL: Accelerators for Model Innovation
To enable operators described at the DSL level to truly land on diverse new hardware and achieve near-theoretical peak performance, compilers must undertake a large number of complex and critical optimization tasks in the middle.
The design goal of DSL is to provide a high-level abstract representation of operators, focusing solely on algorithm implementation without needing to consider hardware details. Therefore, what the compiler needs to do is to map a hardware-agnostic code implementation to the acceleration instructions required by different hardware. It must not only perform fine-grained fusion optimizations on operator logic, such as affine transformations, tiling, fusion, and unrolling, to construct scheduling structures that conform to hardware execution models, but also complete automatic data layout rearrangements, including layout transformations, packing, memory swizzling, etc., to maximize cache hit rates. Meanwhile, the compiler must manage different memory hierarchies of hardware, and it must automatically vectorize high-level tensor operations and further tensorize them to specific hardware’s matrix units or dedicated instructions to fully leverage computing power. Register allocation and overflow management are also crucial, as the compiler needs to find the best balance between high register pressure and memory access overhead. Finally, through an automatic tuning system, the compiler can automatically explore tile sizes, parallel strategies, and memory access patterns in a vast scheduling search space, achieving stronger performance solutions than manual tuning. Relying on this entire set of optimization capabilities, DSL truly possesses the vitality for cross-hardware, high performance, and rapid evolution.
Connecting to Diverse Hardware: A Unified Entry in the RISC-V Era
For a long time, the AI software stack has been constrained by the CUDA ecosystem, where all high-performance optimizations had to be manually written in GPU kernels. However, the current hardware forms are completely different: TPU, NPU, XPU, DSP, Tenstorrent’s RISC-V DSA, various customizable DSAs, edge NPUs, cloud ASICs, and continuously evolving FPGA accelerators… The types of hardware are growing exponentially, and the architectural differences are enormous. In the face of such an ecosystem, it is impossible for the software layer to rebuild an entire set of operator libraries for every hardware; the traditional reliance on handwritten kernels can no longer be sustained.
Therefore, compilers have become the only feasible entry point for new hardware to enter the AI ecosystem. Open ISA architectures like RISC-V can carry universal computing semantics through a unified IR intermediate representation and freely extend based on this; high-level operator DSLs can be automatically lowered by the compiler to the optimal scheduling and instruction sequences required by different hardware; the same operator description can also cross differentiated instruction sets, parallel modes, and memory topologies, achieving automatic adaptation and optimal code generation. This means that new hardware no longer needs to “reinvent CUDA” to quickly support mainstream AI models, truly gaining the capability to enter the ecosystem.
This is especially critical for RISC-V DSA. The customizable ISA of RISC-V, extensible vectors and matrix units, the flexible on-chip SRAM architecture of DSA, instruction extensions for multi-card communication, and heterogeneous acceleration forms bring unprecedented architectural freedom but also require compilers to take on a more important role. They must complete a series of hardware-specific optimizations based on all the characteristics of the hardware architecture, including instruction selection, vectorize/tensorize matching, tile/blocking planning, automatic memory layout optimization, memory space management, SRAM buffer lifecycle management, software pipeline scheduling, and multi-card communication scheduling, among other deeply hardware-specialized processes. New hardware architectures will bring memory hierarchy designs that differ from traditional CPUs and GPUs, presenting new memory management challenges for compilers, such as how to utilize limited SRAM; the widely used multi-accelerator design of AI accelerator cards also brings new tile strategy changes, where the parallel capabilities of common AI accelerator cards are superior to CPUs but slightly weaker than GPUs, leading to different operator block partition sizes; and various optimizations that fully utilize hardware characteristics, including asynchronous load/store hardware implementations that require software pipeline scheduling optimizations to fully leverage hardware features. Only a strong compiler infrastructure can truly transform these hardware extensibilities into performance dividends, fully leveraging the characteristics of different hardware. Otherwise, the potential of RISC-V in the AI era will remain untapped.
5. Operator Libraries × Compilers × RISC-V DSA: Moving Towards an Open AI Infrastructure
In the previous sections, we have seen that the emergence of operator DSL has decoupled operator development from hardware, providing unprecedented opportunities for new hardware to enter the AI market. In the past, any operator wanting to run on new hardware required a large amount of repetitive manual adaptation work, including rewriting kernels, redesigning tiling strategies, adjusting memory layouts, mapping threads, and completing a series of performance tuning logic, which was almost equivalent to recreating a CUDA ecosystem. Therefore, in the past decade, very few new hardware vendors have been able to truly break through CUDA’s monopoly. Now, with the maturity of universal operator DSLs, developers can conduct algorithm development on a hardware-agnostic operator abstraction layer, while the compiler is responsible for mapping operators to different hardware architectures. This model allows RISC-V DSA to no longer be bound by the enormous software ecosystem costs, thus gaining feasibility to enter the AI market.
The emergence of FlagGems further validates the feasibility of this model. FlagGems is a high-performance universal operator library implemented using Triton, aimed at providing a series of operators applicable to the PyTorch framework for large language models, accelerating model inference and training. As a new generation of operator libraries, it offers a set of hardware-agnostic operator implementations, solving the problem of hardware vendors repeatedly implementing operator development. It focuses solely on algorithm logic, while the underlying performance optimization and hardware adaptation are entirely handled by the Triton compiler and AutoTuning system. Operator code can be directly migrated to the RISC-V DSA backend without the need to rewrite libraries or manually tune for each new hardware. Developers are liberated from cumbersome low-level implementations, significantly reducing software complexity and greatly improving R&D efficiency. This model not only changes the way operators are developed but also provides a solid foundation for the future openness of hardware ecosystems.
In this new ecosystem, the collaborative role of operator libraries, compilers, and RISC-V DSA is particularly important. Operator libraries provide standardized algorithm implementations, while compilers are responsible for mapping these algorithms to the instructions and execution units of different hardware, achieving automated scheduling and optimization. RISC-V DSA provides an open and extensible execution platform, allowing algorithm logic to run efficiently on various hardware. Combined with Triton’s open-source operator library FlagGems, we envision an ideal future form: developers only need to use DSL to write high-level operator logic, and the compiler generates high-performance executable code tailored to different DSAs through automatic tiling, vectorization, memory coalescing, and AutoTuning optimizations, achieving “one operator, multi-platform reuse.” This process drastically reduces the cost for new hardware vendors to build software ecosystems and significantly shortens the prototype validation cycle for new operators. For AI models,
Of course, potential challenges for DSL still exist. First, Triton is not completely hardware-neutral; it currently only supports NVIDIA and AMD backends, and it can be said that Triton was originally designed for NVIDIA GPUs. Although Triton is open-source, its community development is constrained by OpenAI’s control over the product roadmap, and contributions from external organizations face numerous obstacles. Additionally, the different versions of other hardware vendors have various limitations and trade-offs, leading to significant fragmentation. This results in Triton’s syntax not being entirely aligned with the characteristics of DSA hardware; some syntactic features are still designed for GPUs, which also means that Triton may not necessarily be the final form of operator DSL. In the next article, we will detail the currently flourishing development of DSLs, and only time will tell what the ultimate form of operator computation will be.
In terms of unleashing hardware performance, there are still many new challenges that need to be overcome. For example, Triton’s core IR design needs to support diverse DSA backends, including memory hierarchy configurations, operator instruction extensions, and AutoTuning parameters, while the MLIR Dialect also needs to uniformly describe matrix instructions, vectorization strategies, SPM/SRAM management, pipeline scheduling, and operations of special functional units. Furthermore, automatic instruction selection (Auto-ISA Selection) combined with performance models and AutoTuning needs to automatically select the best computation strategy based on different instruction sets and hardware characteristics, including dot versus matmul, pack versus nopack, low-precision versus high-precision operations, and tensorcore-like versus SIMD unit mappings. These research directions will directly determine the performance limits of operators on RISC-V DSA. In the face of these challenges, deep software-hardware collaborative optimization between compilers and RISC-V DSA is required to fully leverage the performance advantages of hardware.
Overall, the AI software stack is undergoing a transformation from CUDA monopoly to an open computing power platform. FlagGems and Triton have laid the foundation for the new hardware ecosystem, and RISC-V DSA is expected to become an important force in the next generation of AI accelerators. In subsequent articles, we will gradually delve into the design, usage, and discussion of Triton, exploring its advantages and disadvantages, while investigating the performance and future potential of the combination of Triton and RISC-V. In the future, we will also continue to explore the deep integration of different AI compilers with RISC-V DSA, enabling developers to quickly validate algorithms on any hardware and fully unleash the computing potential of new hardware, thus achieving a truly open, flexible, and high-performance AI infrastructure.
——–END——–
Click to read the original article and join 1nfinite
