Huawei’s New Technology
According to the Science and Technology Innovation Board Daily, Huawei will release an AI SSD on August 12, which reduces the reliance of AI inference on HBM technology. The core principle is to efficiently utilize multi-level KV-Cache cache data that occupies HBM/DRAM, as well as persistently store long sequence text (to SSD), which is incremental rather than a replacement. In terms of packaging, it adopts a unique die on board (DOB) stacking, with a single chip expected to stack up to 36 layers to achieve 36TB, and a single hard drive capacity expected to increase to nearly 256TB.
Previously, when analyzing the iteration of NVIDIA’s computing power chips, a clear characteristic was observed: chip upgrades are accompanied by improvements in HBM storage technology. HBM storage technology has improved.
Compared to traditional storage products, HBM uses 3D stacking technology (such as TSV silicon through holes) to vertically stack multiple DRAM chips, significantly enhancing bandwidth and capacity. The bit width of HBM (1024-bit) is four times that of GDDR5 (32-bit), achieving more than three times the bandwidth at the same power consumption, while reducing power consumption by 50%, thus solving the “memory wall” problem in AI computing.
For example,
H100: Equipped with 80GB HBM3, bandwidth 3.35TB/s.
H200: Upgraded to 141GB HBM3e, bandwidth 4.8TB/s (an increase of 43%), doubling the inference performance of the Llama-70B model.
B200: HBM3e memory: uses 8-Hi (8-layer stacking), with a single die capacity of 192GB and bandwidth of 8TB/s.
B300: HBM3e memory: upgraded to 12-Hi (12-layer stacking), with a single die capacity increased by 50% to 288GB, maintaining a bandwidth of 8TB/s.
From the iteration of NVIDIA’s computing power chips, HBM technology has also played a crucial role.
Due to the high growth demand for HBM from computing hardware, the HBM market size is expected to grow from $560 million in 2023 to an estimated $35 billion by 2025, and will exceed $100 billion in the coming years, making it the fastest-growing segment in the semiconductor industry.
Globally, HBM storage is basically monopolized by overseas manufacturers, with Samsung, Hynix, and Micron holding absolute leadership positions.
Currently, domestic HBM technology is still in the catching-up stage, and the domestic computing power chips are a limiting factor. The “storage instead of computing” solution allows the vector data required for AI inference to be moved from DRAM to SSD, significantly improving computing efficiency, reducing reliance on HBM technology, and lowering inference costs.
Principle of “Storage Instead of Computing”
Under normal circumstances, during large model inference, data has to take a long route: first moving from the hard disk (ROM) to the chip’s peripheral memory, then reading into the CPU’s SRAM for computation, with intermediate results passing through cache and DRAM, finally reaching the SSD hard drive, and this process has to be repeated until a result is produced.
However, “Storage Instead of Computing” is different; it stores the intermediate results of inference in advance in the SSD—because SSDs are non-volatile, data will not be lost even during power outages. This way, inference can directly call the data without real-time computation, avoiding the issue of data in memory (like HBM/DRAM, which loses data during power outages) being overwritten, leading to fragmented conversations.
How is it specifically implemented?
First, an external vector knowledge base is added to the model, converting the enterprise’s massive private domain data into multi-dimensional vectors stored in the SSD in advance, solving the immediacy problem of large models “not remembering new things”; after all, SSDs can reliably store data without being easily overwritten like memory.
Second, a three-layer caching mechanism is established: HBM→DDR→SSD, where the capacity of SSD can be thousands of times larger than HBM (generally 32 – 128GB) and DDR, reaching TB levels, specifically storing long sequence content generated during inference and KV cache for multi-turn dialogue generation, thus not occupying too much memory and avoiding slow inference speed due to insufficient memory.
If it indeed aligns with expectations, the domestic storage direction has a beneficial logic; however, this news has already been disclosed by multiple sources yesterday, and the market did not respond significantly today. The specifics will depend on the details presented at tomorrow’s press conference.
In addition, according to the latest survey by TrendForce, the contract price of consumer-grade DDR4 surged by over 60-85% in July, leading to a significant upward revision of the third-quarter consumer-grade DDR4 contract price to an increase of 85%-90%. The increase in LPDDR4X in the third quarter is the largest single-quarter increase in nearly a decade, and storage prices continue to rise sharply, which is the biggest logic for the industry chain to benefit from price increases.
The biggest problem in the storage industry is actually the imbalance in supply and demand for low-end storage and overcapacity, while high-end storage is still in the rising stage. Whether the industry can reverse largely depends on the pace at which the three major manufacturers exit low-end capacity.
Domestic Storage Industry Chain:
Storage IC design: Zhaoyi Innovation, Beijing Junzheng, Dongxin Co., Fudan Microelectronics, Unisoc, Puran Microelectronics, Jucheng Technology, Hengshuo Technology, etc.;
Storage interconnect, control chips, etc.: Lanke Technology, Lianyun Technology;
Storage modules: Demingli, Baiwei Storage, Langke Technology, Jiangbolong, Wanrun Technology;
Packaging and testing: Deep Technology;
System-level solutions: Tongyou Technology, etc.