Jensen Huang stated that NVIDIA’s Blackwell system sales have “far exceeded expectations,” but analysts believe that custom artificial intelligence chips (ASICs) will grow rapidly.
These smaller, cheaper, and more focused AI chips are being developed in-house by companies such as Google, Amazon, Meta, Microsoft, and OpenAI.
Google’s TPU is a leader in the field of AI-specific integrated circuits (ASICs), and some believe these chips are technically on par with NVIDIA’s GPUs, if not superior.
From Google to all other major hyperscale data centers, all major hyperscale data centers are designing custom ASICs (Application-Specific Integrated Circuits).
From TPU to Amazon’s Trainium and OpenAI’s collaboration with Broadcom, these chips are smaller, cheaper, more accessible, and can reduce reliance on NVIDIA GPUs.
The growth rate of custom ASIC chips in the coming years may even surpass that of the GPU market.
In addition to GPUs and ASICs, there are also Field Programmable Gate Arrays (FPGAs), which can be reconfigured via software after manufacturing for various applications such as signal processing, networking, and AI.
Moreover, there is a whole suite of AI chips that support edge AI rather than cloud-based AI. Qualcomm and Apple are heavily investing in these edge AI chips.
GPUs for General Computing
GPUs were originally used primarily for gaming, but as their use shifted to AI workloads, NVIDIA became the world’s most valuable publicly traded company. Over the past year, NVIDIA shipped approximately 6 million of the latest generation Blackwell GPUs.
The shift from gaming to AI began around 2012 when researchers used NVIDIA’s GPUs to build AlexNet, which many consider the “big bang” moment for modern AI.
AlexNet was a tool that participated in a significant image recognition competition. Unlike other competitors that used CPUs, AlexNet achieved remarkable accuracy by relying on GPUs, completely defeating all competitors.
The creators of AlexNet discovered that the parallel processing techniques used by GPUs to render realistic graphics were equally applicable to training neural networks, where computers learn from data rather than relying on code written by programmers. AlexNet showcased the immense potential of GPUs.
Today, GPUs are typically used in conjunction with CPUs and installed in server rack systems deployed in data centers to run cloud-based AI workloads.
CPUs have a few powerful cores for executing sequential general tasks, while GPUs have thousands of smaller cores that focus more on parallel mathematical operations, such as matrix multiplication.
Because GPUs can perform multiple operations simultaneously, they are well-suited for the two main stages of AI computation: training and inference. The training process teaches AI models to learn patterns from large amounts of data, while the inference process uses AI to make decisions based on new information.
GPUs are the main products for general computing from NVIDIA and its major competitor AMD.
Software is the main difference between the two GPU leaders. NVIDIA GPUs are deeply optimized around the proprietary software platform CUDA, while AMD GPUs primarily use an open-source software ecosystem.
AMD and NVIDIA sell GPUs to cloud service providers like Amazon and Microsoft.
Google, Oracle, and CoreWeave then rent GPUs to AI companies by the hour or minute.
For example, the $30 billion agreement between Anthropic and NVIDIA and Microsoft includes 1 gigawatt of NVIDIA GPU computing power. AMD has also received large orders from OpenAI and Oracle.
NVIDIA also sells products directly to AI companies, such as a recent agreement with OpenAI to sell at least 4 million GPUs; additionally, NVIDIA sells products to foreign governments, including those of South Korea, Saudi Arabia, and the UK.
A server rack containing 72 Blackwell GPUs costs about $3 million, with a weekly shipment of about 1,000 units.
NVIDIA’s Senior Director of AI Infrastructure, Harris, stated that he could not have imagined such demand when he joined NVIDIA eight years ago.
“When people talk about building a system with 8 GPUs, they think it’s a bit overkill.”
Dedicated Integrated Circuits for Custom Cloud AI
In the early stages of large language model development, GPU training was crucial, but as models matured, inference became increasingly important. Inference can be performed on lower-performance chips designed for specific tasks. This is where ASIC chips come into play.
GPUs are like a Swiss Army knife, capable of performing various parallel mathematical operations for different AI workloads, while ASICs are like a single-purpose tool.
They are very efficient and fast, but they are designed to perform specific mathematical operations for one type of task.
“Once they are etched into silicon, you cannot change them, so there is a trade-off in flexibility.”
NVIDIA GPUs have enough flexibility to be adopted by many AI companies, but they can cost up to $40,000 and are difficult to obtain.
Nevertheless, startups still rely on GPUs because the upfront costs of designing custom ASICs are higher, starting at tens of millions of dollars.
Analysts say that for large cloud service providers capable of bearing the costs of custom ASIC chips, they are worth it in the long run.
They hope to have more control over the workloads they build.
Meanwhile, they will continue to work closely with NVIDIA and AMD, as they also need capacity.
The demand is so high that it is hard to meet.
Google was the first major tech company to accelerate AI with custom ASICs, launching the Tensor Processing Unit (TPU) in 2015.
Google stated that it considered manufacturing TPUs as early as 2006, but it wasn’t until 2013 that the demand became urgent as AI was set to double the number of data centers.
In 2017, TPUs also contributed to Google’s invention of the Transformer architecture, which supports nearly all modern AI.
Ten years after the launch of the first TPU, Google released the seventh generation TPU in November.
Anthropic announced it would use up to 1 million TPUs to train the LLM Claude model.
Some believe TPUs are technically on par with NVIDIA’s GPUs, if not superior.
“Traditionally, Google has only used them for internal purposes. But many speculate that in the long run, Google may open up TPU usage more widely.”
After acquiring the Israeli chip startup Annapurna Labs in 2015, Amazon Web Services (AWS) became the next cloud service provider to develop its own AI chips.
AWS released Inferentia in 2018 and Trainium in 2022.
AWS is expected to release the third generation of Trainium as early as December.
Ron Diamant, the chief architect of Trainium, stated that Amazon’s ASIC chips are 30% to 40% more cost-effective than those from other hardware suppliers.
“Over time, we have found that Trainium chips meet the needs of both inference and training workloads very well.”
Anthropic is using 500,000 Trainium2 chips to train its models.
Other AWS data centers are equipped with NVIDIA GPUs to meet the needs of AI customers like OpenAI.
Manufacturing ASICs is not easy. This is why companies turn to chip design firms like Broadcom and Marvell.
They provide intellectual property, expertise, and networks to help customers build ASICs.
“So you can see that Broadcom has become one of the biggest beneficiaries of the AI boom.”
Broadcom helped Google build the TPU chips.
Starting in 2026, they will help OpenAI build custom ASICs.
Microsoft has also begun to venture into the ASIC field, with its self-developed Maia 100 chip currently deployed in data centers in the eastern United States. Other manufacturers include Qualcomm and Intel.
A whole host of startups are fully committed to developing custom AI chips, including Cerebras, which produces giant full-wafer AI chips, and Groq, which focuses on inference.
In China, Huawei, ByteDance, and Alibaba are all producing custom ASIC chips, but export controls on advanced equipment and AI chips pose challenges.
Edge AI Based on NPU and FPGA
The last major category of AI chips is designed to run on devices rather than in the cloud.
These chips are typically integrated into the main system-on-chip (SoC) of the device. Edge AI chips, as the name suggests, enable devices to have AI capabilities while also helping save battery power and space to accommodate other components.
“You can perform these operations directly on your phone with very low latency, so there is no need to communicate with the data center, and you can protect your data privacy on your phone.”
Neural Processing Units (NPUs) are one of the main types of edge AI chips.
Qualcomm, Intel, and AMD are all producing NPUs that enable personal computers to have AI capabilities.
While Apple does not use NPUs, the M-series chips in its MacBook computers do include a dedicated neural engine.
Apple has also integrated a neural network accelerator into the latest A-series chips in its iPhones.
“This is very efficient for us and very responsive. We know we have more control over the user experience.”
The latest Android phones also have NPUs built into the main Qualcomm Snapdragon chips, and Samsung’s Galaxy phones are also equipped with NPUs.
NPUs from companies like NXP and NVIDIA power embedded AI in cars, robots, cameras, smart home devices, and more.
“Most of the funding has flowed to data centers, but over time, this will change as we deploy AI to our phones, cars, wearables, and various other applications, with a much higher degree of application than today.”
Additionally, there are also Field Programmable Gate Arrays (FPGAs), which can be reconfigured via software after manufacturing.
While FPGAs are much more flexible than NPUs or ASICs, their raw performance and energy efficiency are lower for AI workloads.
In 2022, AMD acquired Xilinx for $49 billion, becoming the largest FPGA manufacturer; Intel ranks second after acquiring Altera for $16.7 billion in 2015.
These companies designing AI chips all rely on TSMC.
TSMC is building a massive chip manufacturing plant in Arizona, and Apple has committed to shifting some chip production there.
In October, NVIDIA also announced that Blackwell GPUs are now fully produced in Arizona.
Despite the fierce competition in the AI chip space, it is not easy to shake NVIDIA’s dominance.
They have achieved their current position through hard work and years of accumulation.
They have won the developer ecosystem.