Comparing SIMD and Vector Computing: A Historical Perspective

This document compares SIMD (Single Instruction, Multiple Data) and vector computing from the perspective of instruction set architecture (ISA) within the historical context of computer architecture.

Historical Context: From Supercomputers to Desktop Computers

The Era of Vector Computing (1970s-1980s)

The story of large-scale parallel processing began with supercomputers like Cray-1 (1976). At that time, scientists needed to perform the same mathematical operations on massive data arrays for applications such as weather forecasting and physical simulations. The goal was to design a machine capable of continuously “feeding” a very fast and expensive arithmetic unit without letting the CPU become a bottleneck.

This led to the emergence of the vector processor paradigm. The core idea was to allow a single instruction to operate on an entire data “vector,” with the length of these vectors being variable at runtime.

Instruction Set Philosophy: “One operation, process massive data.”
Core Objective: Maximize throughput for large-scale scientific computing workloads.

The “Packed” SIMD Era (1990s)

In the mid-1990s, the focus of computing shifted. General-purpose CPUs in personal computers needed to handle new multimedia workloads: graphics, audio, and video. These data were typically small-sized, fixed-size blocks (e.g., 8-bit pixels in RGB values, 16-bit audio samples).

Designing a complete vector processor within a standard CPU was too complex and expensive. This gave rise to the “Packed” SIMD pragmatic solution. Manufacturers did not create new, flexible vector hardware but instead widened existing floating-point registers and arithmetic logic units (ALUs) to process multiple smaller data points at once.

Instruction Set Philosophy: “Pack small data into a large register and operate on all data at once.”
Key Cases: Intel’s MMX (1997) technology and the later Streaming SIMD Extensions (SSE) (1999).

Comparison of Instruction Sets and Architectures

The divergences in this history created two distinctly different instruction set styles and brought unique architectural trade-offs.

Feature	Vector Computing (e.g., Cray, RISC-V ‘V’)	Packed SIMD (e.g., Intel SSE/AVX, ARM NEON)
Vector Length	Variable. Set at runtime via a special register `<span><span>VL</span></span>` (vector length). The same instruction can process 10 elements or 1000 elements.	Fixed. Vector length is fixed by register size and data type, directly encoded in the instruction. For example, executing the `<span><span>paddb</span></span>` instruction on a 128-bit register always operates on 16 bytes.
Instruction Set Example	`<span><span>vsetvl t0, a0</span></span>` (set vector length)`<span><span>vadd.vv v1, v2, v3</span></span>` (vector addition)	`<span><span>paddb xmm1, xmm2</span></span>` (add 16 bytes)`<span><span>addps xmm1, xmm2</span></span>` (add 4 single-precision floats)
Instruction Set Scalability	Excellent. The same compiled binary can run on machines with 128-bit wide vector units or 1024-bit wide vector units. The code does not need to change.	Poor. Code written for 128-bit SSE registers cannot utilize AVX’s 256-bit registers or AVX-512’s 512-bit registers. You must recompile or write new code paths to take advantage of wider hardware.
Hardware Design	Time parallel. The CPU “fires and forgets,” with vector units processing elements sequentially over multiple cycles in a deeply pipelined ALU.	Space parallel. Hardware is physically wider. A 128-bit ALU operates on all elements in a 128-bit register in one clock cycle.
Memory Access	Flexible. The instruction set includes complex memory operations, such as strided loads (accessing every Nth element) and gather/scatter (accessing irregular memory locations).	Initially rigid. Early SIMD only had simple contiguous memory loads. Although more advanced features have been added over time, its memory model is often less expressive than that of true vector instruction sets.
State and Context Switching	Heavier. The processor must save/restore more state, including `<span><span>VL</span></span>` and vector type registers (`<span><span>VTYPE</span></span>`), making context switching more complex.	Lighter. State mainly consists of the data registers themselves, making it easier to integrate into existing operating systems.

Modern Fusion: The Return of Vector Computing

For a long time, “Packed” SIMD was the dominant paradigm in general-purpose CPUs. However, its “instruction set scalability” issue became a major obstacle, leading to “instruction set explosion,” where new instructions were constantly needed to support wider registers.

This led to a revival of the originally more elegant concept of vector computing. The most prominent example is the RISC-V “V” (Vector) extension. It is a modern interpretation of the classic Cray vector architecture, adopting a flexible, variable-length vector approach. This addresses the long-standing code portability and scalability issues that plagued “Packed” SIMD, proving that the original supercomputer concept is extremely important for the future of general computing, especially for AI and machine learning workloads.

Conclusion

Vector computing is a powerful, flexible, and scalable approach derived from supercomputing.
Packed SIMD is a pragmatic, simpler adaptation that historically dominated, bringing parallel computing capabilities for multimedia to the masses.
In the future, as RISC-V demonstrates, there is a tendency to return to the elegance and scalability of the original vector model.

References

[1] Russell, R. M. (1978). “The CRAY-1 computer system.” Communications of the ACM, 21(1), 63-72.

[2] Intel Corporation. (1997). “MMX Technology Technical Overview.”

[3] Intel Corporation. (1999). “Streaming SIMD Extensions.” Developer.intel.com.[4] RISC-V International. (2021). “RISC-V Vector Extension Specification”, Version 1.0.

Appendix: Combining Vector and Dataflow Architectures for AI/HPC

Combining vector and dataflow architectures is a fundamental concept behind many modern AI accelerators and a major development direction for future HPC systems. This combination creates a powerful synergy, where each architecture addresses different parts of the AI training problem.

Dataflow architecture handles the macro “what to do and when to do it”..
Vector architecture handles the actual numerical computations of micro “how to do it”..

Synergy: Dataflow as the Graph, Vector as the Engine

The structure of deep neural networks is a large computational graph, where layers are nodes and tensors are the data flowing between them.

Role of Dataflow (Macro Architecture): Dataflow architecture is the perfect model for executing such graphs. Execution is driven by the availability of data rather than sequential instruction fetching by the program counter. A node (e.g., convolution layer) “triggers” immediately when all its input tensors are available. This model naturally reveals the massive parallelism in neural networks and transforms the entire network into a highly parallel database.
Role of Vector (Micro Architecture): When a dataflow node “triggers,” it must perform large-scale linear algebra operations. This task is assigned to vector processing units. A single vector instruction can operate on thousands of data elements, perfectly matching the structure of tensors. Vector processors are highly efficient for these dense, regular arithmetic operations, making them more energy-efficient than scalar or SIMD processors when handling these tasks.

Analogy: Factory Assembly Line

Dataflow is the layout of the factory workshop. A workstation only starts working when it receives parts from the previous workstation.
Vector is the advanced robotic arms that efficiently and in parallel execute the actual work at the workstation, then send the parts to the next workstation.

Advantages of the Combination

Revealing Massive Parallelism: Dataflow naturally maps the parallelism of the neural network graph onto the hardware.
Extremely High Computational Efficiency: Vector units perform core mathematical operations with minimal instruction decoding overhead.
Hiding Latency: When one vector unit is busy, the dataflow network can activate other independent vector units.
Specialization: The combined architecture is purpose-built for AI, stripping away unnecessary complexities of general-purpose CPUs.

Real-World Examples

This architectural concept is already at the core of many successful AI accelerators:

Google’s TPU: Its core pulsating array is a hardware embodiment of dataflow principles.
Cerebras Wafer Scale Engine: Maps the neural network graph onto a large core grid on the chip. The communication network is a dataflow system, with each core acting as an execution engine.
SambaNova Systems: Explicitly markets its products as “reconfigurable dataflow architecture.”

In summary, combining vector and dataflow principles can be considered the dominant architectural paradigm for building dedicated, high-performance, and energy-efficient AI hardware today.

While many RISC-V AI accelerators support RV64F (64-bit single-precision floating point), this does not make them the key feature of being an “AI accelerator”.

The performance of these chips comes from large-scale parallel computing, typically achieved through the following:

RISC-V ‘V’ (Vector) extension..
Integrating a large number of simple cores in a grid (a kind of dataflow approach).
Proprietary custom extensions designed for AI workloads.

The scalar RV64F instruction set is only used by the main control core for general-purpose tasks, not for the core matrix and vector mathematics required by AI.

Here are some outstanding examples of RISC-V-based AI accelerators that are on the market or close to market, along with details about their software stacks.

1. Esperanto Technologies

Product:ET-SoC-1 “on-chip supercomputer.”
Hardware Architecture: Contains 1088 energy-efficient 64-bit RISC-V cores (ET-Minion), each with its own vector/tensor unit. It also has four high-performance 64-bit RISC-V cores (ET-Maxion) for control. Its instruction set is RV64GCV.
Compiler and Software Stack:

High-Level Framework: Users start with models from standard frameworks (such as PyTorch, TensorFlow, or ONNX).
Graph Compiler: Esperanto provides a proprietary graph compiler that takes models from frameworks and optimizes them for its specific chip architecture. It performs operations like layer fusion and maps the neural network graph onto over 1000 cores.
Low-Level Toolchain: The toolchain is based on LLVM and GCC, fully supporting its RISC-V cores and vector extensions. This is used to compile the low-level “core functions (kernel)” that execute the actual mathematical operations for each layer.
Core Function Library: They provide a highly optimized AI function library (convolution, matrix multiplication, activation functions, etc.) fine-tuned for the vector units of the ET-Minion cores. The graph compiler targets these optimized function libraries for compilation.
Runtime: A complex software runtime manages the chip, coordinating model loading, data movement between off-chip memory and core local SRAM, and synchronization among over 1000 cores.

2. Tenstorrent

Product:Grayskull and Wormhole AI accelerator cards.
Hardware Architecture: These chips feature a “Tensix” core grid. Each core is a custom processor, including a small but powerful RISC-V CPU and multiple dedicated engines for tensor mathematics (SIMD and matrix multiplication). The RISC-V core is used for local control, orchestration, and handling complex operations.
Compiler and Software Stack:

TT-Buda: This is their graph compiler and runtime. It receives a PyTorch graph, partitions it, and maps it onto the Tensix core grid on one or more chips. It handles the entire compilation and execution process.
TT-Metal: This is their low-level, “bare metal” programming model, conceptually similar to NVIDIA’s CUDA. It allows developers to write custom C++ code directly to program the RISC-V cores and tensor engines, providing them with fine-grained control over the hardware for maximum performance or to implement novel operations.

High-Level Framework: Fully supports PyTorch (via torch.compile) and TensorFlow.
Software 2.0 Software Stack: Their software stack is designed to be highly dynamic and developer-friendly.
Low-Level Toolchain: The software stack is built on a standard LLVM/Clang toolchain and extended to support their custom RISC-V cores.

3. SiFive (IP Supplier)

Product:SiFive Intelligence™ X280 CPU IP. This is not a chip but a core licensed for other companies to build their own AI chips.
Hardware Architecture: The X280 is a 64-bit multi-core processor that fully supports RISC-V ‘V’ (Vector) extension v1.0. It also includes SiFive Intelligent Extensions, which are custom instructions designed for further accelerating AI workloads (e.g., matrix multiplication).
Compiler and Software Stack:

Ecological Approach: Since SiFive provides IP, they focus on empowering a broad open-source software ecosystem.
High-Level Framework: Models are introduced through frameworks like TensorFlow Lite or ONNX.
Graph Compiler/Inferences Engine: The X280 is supported by open-source projects like Apache TVM, Google’s IREE, and Plumerai’s Larq. These tools can compile AI models into code that runs efficiently on the X280’s vector engine.
Low-Level Toolchain: SiFive provides and contributes to a standard GCC and LLVM toolchain that has strong support for RISC-V vector extensions.
Core Function Library: SiFive provides vcl (vector computing library) and collaborates with the community to develop libraries like XNNPACK to ensure optimized core functions are available for vector extensions.

In summary, no serious AI accelerator is solely based on RV64F. They all rely on advanced features like vector extensions or custom hardware, supported by complex multi-layer software stacks designed to convert high-level Python models into highly optimized code for their specific parallel architectures.

Recalling a friend’s words, “Americans have no need for new CPU architectures,” RISC-V still thrives as an accelerator concept. The accelerators of yesteryear have become the cornerstone of general computing under the influence of AGI. However, the requirements for an ecosystem are even higher for accelerators than for CPUs, as there is no “Linux” without accelerators. OpenCL has already been squeezed out by CUDA.