Bus – HBM2E

I am pleased to provide you with a detailed introduction to the HBM2E bus.

HBM2E (High Bandwidth Memory 2E) is a significant milestone in the development of high bandwidth memory technology. To understand HBM2E, it is essential to first grasp its foundation – HBM technology.

1. What is HBM? – Technical Background

Traditional GPUs and CPUs use GDDR or DDR memory located on the motherboard, communicating via traces on the PCB. This method encounters physical bottlenecks in speed and capacity (such as signal integrity, power consumption, and area).

HBM is a revolutionary 3D stacked memory technology, with the core idea being:

• 3D Stacking: Stacking multiple DRAM chips (such as 8-layer or 12-layer) vertically, like a stack, using Through-Silicon Via (TSV) technology.

• Close to the Processor: The HBM stack is placed right next to the computing cores like GPUs or CPUs on the same substrate through an Interposer, which acts like a miniature “silicon highway,” enabling ultra-short, high-density interconnections.

• Wide Interface: Unlike the narrow and fast interfaces of traditional memory (such as 32 bits), HBM uses extremely wide and relatively slower interfaces (such as 1024 bits/stack). Bandwidth = Frequency × Bit Width, and the enormous bit width compensates for the lower frequency, achieving extremely high total bandwidth.

2. Positioning and Key Features of HBM2E

HBM2E (also known as HBM2e or HBM2 Enhanced) is an enhanced version of HBM2 standardized by JEDEC (Solid State Technology Association) at the end of 2018. It is not the next generation HBM3, but rather an optimization and refinement of HBM2, designed to meet the explosive growth in memory performance and capacity demands in fields such as AI and HPC.

Its core features and enhancements are primarily reflected in the following aspects:

a. Extremely High Bandwidth

• Data Rate: HBM2E significantly increases the data transfer rate per pin from approximately 2.4 Gbps in HBM2 to a maximum of 3.6 Gbps, with some implementations by manufacturers even reaching 3.8 Gbps or higher.

• Single Stack Bandwidth: Under a standard 1024-bit wide interface:

• @ 3.2 Gbps: 1024 bits * 3.2e9 / 8 ≈ 410 GB/s (common value)

• @ 3.6 Gbps: 1024 bits * 3.6e9 / 8 ≈ 460 GB/s

• Total Bandwidth: High-end GPUs/accelerators typically integrate 2, 4, or even 6 HBM2E stacks. For example, with 4 stacks, the total bandwidth can easily exceed 1.6 TB/s or even 1.8 TB/s. This enables the processing of extremely large AI models and massive scientific computation data.

b. Larger Capacity

• Stacking Layers: The HBM2E standard supports up to 12 dies (including optional underlying logic dies) per DRAM stack.

• Single Stack Capacity: With the increase in stacking layers and single die capacity (such as 8Gb, 16Gb), a single HBM2E stack can reach 8GB, 16GB, or even 24GB.

• Total Capacity: Similarly, multi-stack configurations can achieve 32GB, 48GB, 64GB, 96GB and other massive on-chip memory capacities. This allows entire large AI models or datasets to be stored directly in the ultra-fast memory close to the processor, avoiding slow data exchanges with system main memory (DDR).

c. Excellent Energy Efficiency

• Despite the extremely high total bandwidth, HBM2E has a low operating voltage (~1.2V), relatively moderate data transfer rates (without needing to reach extremely high frequencies like GDDR6), and very short interconnect distances, resulting in outstanding energy efficiency (bandwidth/watt). This is crucial for data centers facing significant challenges in power consumption and heat dissipation.

3. HBM2E vs. HBM2 vs. HBM3

Feature	HBM2 (1st/2nd Generation)	HBM2E (Enhanced Version)	HBM3 (Next Generation)
Data Rate (per pin)	~2.0 – 2.4 Gbps	up to 3.6 – 3.8 Gbps	up to 6.4 Gbps+
Single Stack Bandwidth	~256 GB/s	up to 460+ GB/s	up to 819 GB/s
Maximum Stacking Layers	8 (4-Hi, 8-Hi)	12-Hi	12-Hi+ (supports higher)
Single Stack Capacity	typically 4GB, 8GB	up to 16GB, 24GB	up to 32GB, 64GB
Channel Bit Width	1024-bit / stack	1024-bit / stack	1024-bit or independent channels
Voltage	1.2V	1.2V	1.1V (lower power)

4. Application Areas

HBM2E is almost exclusively designed for applications that have extreme requirements for memory bandwidth and capacity:

• Artificial Intelligence and Machine Learning (AI/ML): Training extremely large Transformers, large language models (LLMs, such as the GPT series), etc., requires rapid exchanges of hundreds of GB of parameters and gradients in memory.

• High-Performance Computing (HPC): Climate simulation, fluid dynamics, molecular dynamics, and other scientific computations require processing massive amounts of data.

• High-End Graphics Processing: Professional visualization, virtual reality (VR)/augmented reality (AR) servers, top-tier graphics cards.

• Network Data Processing: Advanced DPU/IPU for high-speed packet processing and encryption.

5. Summary of Advantages and Disadvantages

• Advantages:

• Extreme Bandwidth: Far surpassing any existing memory technology.

• Ultra-Large Capacity: Capable of integrating massive memory on-chip.

• High Energy Efficiency: Low power consumption per unit bandwidth.

• Small Physical Area: Vertical stacking saves valuable space on the 2D plane.

• Disadvantages:

• Extremely High Cost: Complex 3D stacking and TSV processes, use of interposers, and more complex packaging (such as CoWoS) lead to very high costs.

• Manufacturing Complexity: Yield challenges and complex supply chains.

• Heat Dissipation Challenges: High-density stacked chips generate concentrated heat sources, requiring advanced cooling solutions.

Conclusion

HBM2E represents a significant leap in HBM2 technology, successfully meeting the stringent memory performance demands of the AI and HPC industries from 2019 to 2022 before the full adoption of HBM3. It is a key engine driving the performance of top-tier accelerators like NVIDIA A100 (40GB/80GB versions) and AMD MI100, laying the hardware foundation for today’s breakthroughs in artificial intelligence.

With the arrival of HBM3 and future HBM4, the competition for bandwidth and capacity will continue, but HBM2E undoubtedly plays a crucial role in this evolutionary path.