In recent years, the concept of FPGA has become increasingly prevalent. For example, Bitcoin mining has seen the use of FPGA-based mining machines. Additionally, Microsoft previously announced that it would use FPGAs “instead of” CPUs in its data centers, among other applications.
In fact, FPGA is not unfamiliar to professionals; it has been widely used for many years. However, most people still do not understand it well and have many questions—What exactly is FPGA? Why should it be used? What are the characteristics of FPGA compared to CPUs, GPUs, and ASICs (Application-Specific Integrated Circuits)? …
Today, with this series of questions, let’s together—uncover the mysteries of FPGA.
1. Why Use FPGA?
As is well known, the Moore’s Law for general-purpose processors (CPUs) has reached its twilight, while the scale of machine learning and web services is growing exponentially.
People are using custom hardware to accelerate common computing tasks, but the rapidly changing industry demands that this custom hardware be reprogrammable to perform new types of computing tasks.
FPGA is precisely a hardware-reconfigurable architecture. Its full name in English is Field Programmable Gate Array.
For many years, FPGA has been used as a small-batch alternative to ASICs, but in recent years, it has been deployed on a large scale in data centers of companies like Microsoft and Baidu, providingboth powerful computing capabilities and sufficient flexibility.
Comparison of performance and flexibility among different architectures
Why is FPGA fast? “It’s all about the good company it keeps.”
Both CPUs and GPUs belong to the von Neumann architecture, which involves instruction decoding execution and shared memory. The reason FPGA is more efficient than CPUs and even GPUs is fundamentally due to its architecture, which is instruction-less and does not require shared memory.
In the von Neumann architecture, since execution units (like CPU cores) can execute arbitrary instructions, there is a need for instruction memory, decoders, various instruction arithmetic units, and branch jump processing logic. Due to the complexity of the control logic for instruction flow, there cannot be too many independent instruction flows, so GPUs use SIMD (Single Instruction Multiple Data) to allow multiple execution units to process different data in sync, and CPUs also support SIMD instructions.
In contrast, the functionality of each logic unit in FPGA is determined during reprogramming (burning), and no instructions are needed.
In the von Neumann architecture, memory serves two purposes: one is to save state, and the other is for communication between execution units.
Since memory is shared, access arbitration is required; to utilize access locality, each execution unit has a private cache, which necessitates maintaining cache consistency among execution components.
For the need to save state, the registers and on-chip memory (BRAM) in FPGA belong to their respective control logic, eliminating unnecessary arbitration and caching.
For the need for communication, the connections between each logic unit in FPGA are determined during reprogramming (burning), and do not require communication through shared memory.
Having discussed these high-level concepts, how does FPGA actually perform? Let’s
look at both compute-intensive tasks and communication-intensive tasks.
Compute-intensive tasks include matrix operations, image processing, machine learning, compression, asymmetric encryption, and sorting for Bing search, among others. These tasks are generally offloaded from the CPU to the FPGA for execution. For these tasks, the current performance of integer multiplication on the Altera (now Intel) Stratix V FPGA is roughly equivalent to that of a 20-core CPU, while the floating-point multiplication performance is comparable to that of an 8-core CPU, but is an order of magnitude lower than that of a GPU. The next generation FPGA, Stratix 10, which we are about to use, will be equipped with more multipliers and hardware floating-point units, theoretically achieving computing power comparable to today’s top GPU computing cards.
Estimated integer multiplication capability of FPGA (not using DSP, based on logic resource usage)
Estimated floating-point multiplication capability of FPGA (float16 using soft core, float32 using hard core)
In data centers, the core advantage of FPGA over GPU is latency.
For tasks like sorting Bing search results, it is essential to return search results as quickly as possible, which requires minimizing latency at every step.
If using a GPU for acceleration, to fully utilize the GPU’s computing power, the batch size cannot be too small, leading to latency in the millisecond range.
When using FPGA for acceleration, only microsecond-level PCIe latency is needed (our current FPGA is used as a PCIe acceleration card).
In the future, after Intel launches Xeon + FPGA connected via QPI, the latency between CPU and FPGA can be reduced to below 100 nanoseconds, making it comparable to accessing main memory.
Why is FPGA’s latency so much lower than that of GPU?
This is fundamentally due to architectural differences.
FPGA possesses both pipeline parallelism and data parallelism, while GPU primarily has data parallelism (with limited pipeline depth).
For example, processing a data packet involves 10 steps; FPGA can build a 10-stage pipeline, with different stages processing different data packets. Once a data packet passes through all 10 stages, it is completed and can be output immediately.
In contrast, the GPU’s data parallel approach involves 10 computing units, each processing different data packets, but all computing units must operate in sync, performing the same task (SIMD, Single Instruction Multiple Data). This requires that 10 data packets must be input and output together, increasing input-output latency.
When tasks arrive one by one rather than in batches, pipeline parallelism can achieve lower latency than data parallelism. Therefore, forstreaming computation tasks, FPGA has an inherent advantage in terms of latency over GPU.
Comparison of orders of magnitude for compute-intensive tasks: CPU, GPU, FPGA, ASIC (using 16-bit integer multiplication as an example, numbers are only estimates of orders of magnitude)
ASIC chips excel in throughput, latency, and power consumption, but Microsoft has not adopted them for two reasons:
-
The computing tasks in data centers are flexible and variable, while ASIC development costs are high and cycles are long. After deploying a batch of acceleration cards for a certain neural network, if another neural network becomes more popular, the investment is wasted. FPGA can update its logic functions in just a few hundred milliseconds.FPGA’s flexibility protects investments; in fact, Microsoft’s current use of FPGA is quite different from its initial concept.
-
Data centers are rented out to different tenants; if some machines have neural network acceleration cards, some have Bing search acceleration cards, and others have network virtualization acceleration cards, task scheduling and server maintenance become complicated.Using FPGA can maintain homogeneity in the data center.
Next, let’s look at communication-intensive tasks.
Compared to compute-intensive tasks, communication-intensive tasks are not very complex in processing each input data; they generally involve simple calculations to produce output, making communication often a bottleneck. Examples of communication-intensive tasks include symmetric encryption, firewalls, and network virtualization.
Comparison of orders of magnitude for communication-intensive tasks: CPU, GPU, FPGA, ASIC (using 64-byte network packet processing as an example, numbers are only estimates of orders of magnitude)
For communication-intensive tasks, FPGA has even greater advantages over CPUs and GPUs.
In terms of throughput, FPGA transceivers can connect directly to 40 Gbps or even 100 Gbps network cables, processing packets of any size at line speed; while CPUs need to receive packets from network cards before processing, many network cards cannot handle 64-byte small packets at line speed. Although multiple network cards can be used to achieve high performance, the number of PCIe slots supported by CPUs and motherboards is often limited, and network cards and switches themselves can be quite expensive.
In terms of latency, when the network card receives packets to the CPU, and the CPU then sends them back to the network card, even using high-performance packet processing frameworks like DPDK, the latency can be 4-5 microseconds. A more serious issue is that the latency of general-purpose CPUs is not stable. For example, under high load, forwarding latency can rise to tens of microseconds or even higher (as shown in the figure below); clock interrupts and task scheduling in modern operating systems also add uncertainty to latency.
Comparison of forwarding latency between ClickNP (FPGA), Dell S6000 switch (commercial switch chip), Click+DPDK (CPU), and Linux (CPU), with error bars indicating 5% and 95%. Source: [5]
Although GPUs can also process packets at high performance, GPUs do not have network ports, meaning packets must first be received from the network card before being processed by the GPU. This limits throughput to that of the CPU and/or network card, and the latency of the GPU itself is even more concerning.
So why not implement these network functions in network cards or use programmable switches? The flexibility of ASIC remains a significant drawback.
Although there are increasingly powerful programmable switch chips available, such as Tofino supporting P4 language, ASICs still cannot perform complex stateful processing, such as certain custom encryption algorithms.
In summary,the main advantages of FPGA in data centers are stable and extremely low latency, suitable for both streaming compute-intensive tasks and communication-intensive tasks.
2. Microsoft’s Practice of Deploying FPGA
In September 2016, Wired magazine published an article titled “Microsoft Bets Its Future on FPGA,” detailing the history of the Catapult project.
Subsequently, Doug Burger, the head of the Catapult project, demonstrated FPGA-accelerated machine translation alongside Microsoft CEO Satya Nadella at Ignite 2016.
The total computing power of the demonstration was 1.03 million T ops, equivalent to 1.03 Exa-ops, comparable to 100,000 top GPU computing cards. The power consumption of a single FPGA (along with on-board memory and network interfaces) is about 30 W, only increasing the total server power consumption by one-tenth.
Demonstration at Ignite 2016: 1 Exa-op (10^18) machine translation computing power per second
Microsoft’s deployment of FPGA has not been smooth. The question of where to deploy FPGA has gone through three stages:
-
Dedicated FPGA clusters filled with FPGAs
-
One FPGA per machine, using dedicated network connections
-
One FPGA per machine, placed between network cards and switches, sharing server networks
Three stages of Microsoft’s FPGA deployment, source: [3]
The first stage involved dedicated clusters filled with FPGA acceleration cards, resembling a supercomputer made entirely of FPGAs.
The image below shows the earliest BFB experimental board, which housed 6 FPGAs on a single PCIe card, with 4 PCIe cards inserted into each 1U server.
The earliest BFB experimental board, which housed 6 FPGAs. Source: [1]
Notice the name of the company. In the semiconductor industry, as long as the volume is large enough, the price of chips will tend to the price of sand. It is rumored that it was precisely because this company was unwilling to offer “sand prices” that another company was chosen.
Of course, now both companies’ FPGAs are used in the data center field.As long as the scale is large enough, concerns about FPGA prices being too high will be unnecessary.
The earliest BFB experimental board, with 4 FPGA cards inserted into a 1U server. Source: [1]
The supercomputer-like deployment method means that there is a dedicated cabinet filled with servers like the one shown above, which contain 24 FPGAs (left image).
This method has several issues:
-
FPGAs in different machines cannot communicate with each other, limiting the scale of problems that FPGAs can handle to the number of FPGAs in a single server;
-
Other machines in the data center must concentrate tasks to this cabinet, creating in-cast, making it difficult to achieve stable network latency.
-
The dedicated FPGA cabinet constitutes a single point of failure; if it fails, no one can accelerate anything;
-
Servers with FPGAs are custom-built, complicating cooling and maintenance.
Three methods of deploying FPGA, from centralized to distributed. Source: [1]
A less aggressive approach is to deploy a server filled with FPGAs in each cabinet (shown in the image above). This avoids the issues mentioned in (2) and (3), but (1) and (4) remain unresolved.
The second stage, toensure homogeneity of servers in the data center (which is also an important reason for not using ASICs), involved inserting one FPGA into each server (shown in the right image), with FPGAs connected via dedicated networks. This was also the deployment method adopted in Microsoft’s paper presented at ISCA’14.
Open Compute Server in the rack. Source: [1]
Interior of Open Compute Server. The red box indicates the location for the FPGA. Source: [1]
Open Compute Server after inserting FPGA. Source: [1]
Connection and fixing between FPGA and Open Compute Server. Source: [1]
FPGA uses Stratix V D5, with 172K ALMs, 2014 M20K on-chip memory, and 1590 DSPs. The board has an 8GB DDR3-1333 memory, a PCIe Gen3 x8 interface, and two 10 Gbps network interfaces. The FPGA between cabinets uses dedicated network connections, with one group of 8 10G ports connected in a ring, and another group of 6 10G ports connected in a ring, without using switches.
Network connection method between FPGAs in the cabinet. Source: [1]
This cluster of 1632 servers and 1632 FPGAs has doubled the overall performance of Bing’s search result sorting (in other words, halving the number of servers needed).
As shown in the image below, every 8 FPGAs are connected in a chain, with the aforementioned 10 Gbps dedicated network cable used for communication. Each of these 8 FPGAs has its own role: some are responsible for feature extraction from documents (yellow), some for calculating feature expressions (green), and some for scoring documents (red).
FPGA accelerates the process of sorting Bing search results. Source: [1]
FPGA not only reduces the latency of Bing search but also significantly improves latency stability. Source: [4]
Both local and remote FPGAs can reduce search latency, with the communication latency of remote FPGAs being negligible compared to search latency. Source: [4]
FPGA deployment in Bing has been successful, and the Catapult project continues to expand within the company.
The department with the most servers within Microsoft is the cloud computing Azure department.
The urgent problem for the Azure department is the overhead caused by network and storage virtualization. Azure sells virtual machines to customers, requiring network functions such as firewalls, load balancing, tunnels, and NAT for the virtual machines. Since the physical storage of cloud storage is separated from computing nodes, data must be transported from storage nodes over the network, requiring compression and encryption.
In the era of 1 Gbps networks and mechanical hard drives, the CPU overhead for network and storage virtualization was negligible. As network and storage speeds increase, with networks reaching 40 Gbps and SSDs achieving throughput of 1 GB/s, CPUs are gradually becoming overwhelmed.
For example, the Hyper-V virtual switch can only handle around 25 Gbps of traffic, failing to reach 40 Gbps line speed, and performs worse when packet sizes are smaller; AES-256 encryption and SHA-1 signing can only be processed at 100 MB/s per CPU core, which is only one-tenth of an SSD’s throughput.
Number of CPU cores required for network tunneling protocols and firewall processing at 40 Gbps. Source: [5]
To accelerate network functions and storage virtualization, Microsoft has deployed FPGAs between network cards and switches.
As shown in the image below, each FPGA has a 4 GB DDR3-1333 DRAM, connected to a CPU socket via two PCIe Gen3 x8 interfaces (physically a PCIe Gen3 x16 interface, as the FPGA does not have a x16 hard core, logically treated as two x8). The physical network card (NIC) is just a standard 40 Gbps network card, used solely for communication between the host and the network.
Architecture of Azure servers deploying FPGA. Source: [6]
FPGA (SmartNIC) virtualizes a network card for each virtual machine, allowing virtual machines to access this virtual network card directly through SR-IOV. The data plane functions originally in the virtual switch have been moved to the FPGA, allowing virtual machines to send and receive network packets without CPU involvement or passing through the physical network card (NIC). This not only saves CPU resources that can be sold but alsoimproves the network performance of virtual machines (25 Gbps), reducing network latency between virtual machines in the same data center by ten times.
Accelerated architecture for network virtualization. Source: [6]
This is the third-generation architecture for deploying FPGA at Microsoft, which is currently used for large-scale deployment of “one FPGA per server”.
FPGA’s initial goal of reusing host networks to accelerate networking and storage has a far-reaching impact, extending the network connections between FPGAs to the scale of the entire data center, creating a truly cloud-scale “supercomputer”.
In the second-generation architecture, the network connections between FPGAs were limited to within the same rack, and the dedicated interconnection method between FPGAs was difficult to scale, while forwarding through the CPU incurred too high overhead.
In the third-generation architecture, FPGAs communicate via LTL (Lightweight Transport Layer). Latency within the same rack is under 3 microseconds; within 8 microseconds, it can reach 1000 FPGAs; and within 20 microseconds, it can reach all FPGAs in the same data center. Although the second-generation architecture has lower latency for up to 8 machines, it can only access 48 FPGAs through the network. To support extensive communication between FPGAs, the LTL in the third-generation architecture also supports PFC flow control protocol and DCQCN congestion control protocol.
Vertical axis: LTL latency, horizontal axis: number of reachable FPGAs. Source: [4]
Logical module relationships within FPGA, where each Role represents user logic (such as DNN acceleration, network function acceleration, encryption), and the outer part is responsible for communication between various Roles and between Roles and peripherals. Source: [4]
Data center acceleration plane formed by FPGAs, situated between the network switching layer (TOR, L1, L2) and traditional server software (software running on CPUs). Source: [4]
Through high-bandwidth, low-latency network interconnections, FPGAs form a data center acceleration plane situated between the network switching layer and traditional server software.
In addition to the acceleration of network and storage virtualization required by every server providing cloud services, the remaining resources on the FPGA can also be used to accelerate Bing search, deep neural networks (DNN), and other computing tasks.
For many types of applications, as the scale of distributed FPGA accelerators increases, their performance improvement is super-linear.
For example, in CNN inference, when only one FPGA is used, the on-chip memory is insufficient to hold the entire model, necessitating constant access to model weights in DRAM, creating a performance bottleneck; if the number of FPGAs is sufficient, each FPGA can be responsible for a layer or several features within a layer, allowing the model weights to be fully loaded into on-chip memory, eliminating the DRAM performance bottleneck and fully utilizing the performance of FPGA computing units.
Of course, overly fine-grained partitioning can also lead to increased communication overhead. The key to partitioning tasks across a distributed FPGA cluster is balancing computation and communication.
From neural network models to HaaS on FPGA. Utilizing the parallelism within the model, different layers and features of the model are mapped to different FPGAs. Source: [4]
At the MICRO’16 conference, Microsoft proposed the concept of Hardware as a Service (HaaS), which allows hardware to be treated as a schedulable cloud service, enabling centralized scheduling, management, and large-scale deployment of FPGA services.
Hardware as a Service (HaaS). Source: [4]
From the first generation of dedicated server clusters filled with FPGAs, to the second generation of FPGA acceleration card clusters connected via dedicated networks, to the current large-scale FPGA cloud that reuses data center networks, three guiding principles have shaped our path:
-
Hardware and software are not in a relationship of replacement but rather cooperation;
-
Flexibility is essential, meaning the ability to be software-defined;
-
Scalability is a must.
3. The Role of FPGA in Cloud Computing
Finally, I would like to share my personal thoughts on the role of FPGA in cloud computing. As a third-year PhD student, my research at Microsoft Research Asia attempts to answer two questions:
-
What role should FPGA play in cloud-scale network interconnection systems?
-
How to program heterogeneous systems of FPGA + CPU efficiently and scalably?
My main regret in the FPGA industry is that the mainstream use of FPGA in data centers, from internet giants other than Microsoft to the two major FPGA manufacturers and academia, is mostly treating FPGA as a computing accelerator for compute-intensive tasks, similar to GPUs. But is FPGA really suitable for doing what GPUs do?
As mentioned earlier,the biggest difference between FPGA and GPU lies in their architecture; FPGA is more suitable for low-latency streaming processing, while GPU is better for processing large batches of homogeneous data.
Since many people intend to use FPGA as a computing accelerator, the high-level programming models launched by the two major FPGA manufacturers are also based on OpenCL, mimicking the batch processing model based on shared memory used by GPUs. For a CPU to assign a task to FPGA, it first needs to load it into the DRAM on the FPGA board, then instruct the FPGA to start execution, and finally, the FPGA returns the execution results to DRAM, notifying the CPU to retrieve them.
Why go through the DRAM on the board for efficient communication between CPU and FPGA via PCIe? Perhaps it is an engineering implementation issue; we found that writing to DRAM, starting the kernel, and reading from DRAM in OpenCL takes 1.8 milliseconds. In contrast, communication via PCIe DMA only takes 1-2 microseconds.
Performance comparison between PCIe I/O channel and OpenCL. The vertical axis is on a logarithmic scale. Source: [5]
Communication between multiple kernels in OpenCL is even more exaggerated, with the default method also being through shared memory.
This article begins by stating that FPGA is more efficient than CPU and GPU, and the fundamental advantage of its architecture is that it is instruction-less and does not require shared memory. Using shared memory for communication between multiple kernels is unnecessary in sequential communication (FIFO). Moreover, the DRAM on FPGA is generally much slower than that on GPUs.
Therefore, we proposed the ClickNP network programming framework [5],which uses channels instead of shared memory for communication between execution units (elements/kernels) and between execution units and host software.
Applications that require shared memory can also be implemented on the basis of channels, as CSP (Communicating Sequential Process) and shared memory are theoretically equivalent. ClickNP is still a framework based on OpenCL, limited by the constraints of C language for hardware description (though HLS is indeed much more efficient than Verilog). The ideal hardware description language is probably not C.
ClickNP uses channels for communication between elements, source: [5]
ClickNP uses channels for communication between FPGA and CPU, source: [5]
Low-latency streaming processing requires the most efficient communication.
However, due to the limitations of parallelism and operating system scheduling, CPU communication efficiency is not high, and latency is also unstable..
Moreover, communication inevitably involves scheduling and arbitration; due to the limitations of single-core performance and the inefficiency of inter-core communication, CPU scheduling and arbitration performance is constrained, while hardware is well-suited for such repetitive tasks. Therefore, my PhD research defines FPGA as the “big steward” of communication, whether it is communication between servers, between virtual machines, between processes, or between CPUs and storage devices, all can be accelerated using FPGA.
Success and failure both stem from the same source. The lack of instructions is both an advantage and a weakness of FPGA.
Every time a different task is performed, a certain amount of FPGA logic resources must be occupied. If the tasks to be performed are complex and not highly repetitive, they will occupy a large amount of logic resources, most of which will remain idle. In such cases, it is not as efficient as using a von Neumann architecture processor.
Many tasks in data centers have strong locality and repetitiveness: one part is the network and storage tasks required by virtualization platforms, which belong to communication; the other part is customer computing tasks, such as machine learning and encryption/decryption.
First, use FPGA for what it does best—communication. In the future, it may also be possible to rent FPGA as computing accelerators to customers, similar to AWS.
Whether for communication, machine learning, or encryption/decryption, the algorithms are complex; if one tries to completely replace CPU with FPGA, it will inevitably lead to a significant waste of FPGA logic resources and increase the development cost of FPGA programs. A more practical approach is tohave FPGA and CPU work together, with FPGA handling tasks that have strong locality and repetitiveness, while complex tasks are handled by the CPU.
As we accelerate more and more services like Bing search and deep learning with FPGA; as the data plane of foundational components like network virtualization and storage virtualization is dominated by FPGA; as the “data center acceleration plane” formed by FPGAs becomes a barrier between networks and servers… it seems that FPGA will take control, and the computing tasks on CPUs will become fragmented, driven by FPGA. In the past, we relied on CPUs to offload repetitive computing tasks to FPGAs; will it change in the future to FPGAs taking the lead and offloading complex computing tasks to CPUs? With the advent of Xeon + FPGA, will the ancient SoC be revitalized in data centers?
“Crossing the memory wall, moving towards a programmable world” (Across the memory wall and reach a fully programmable world.)
References:
-
[1] Large-Scale Reconfigurable Computing in a Microsoft Datacenter https://www.microsoft.com/en-us/research/wp-content/uploads/2014/06/HC26.12.520-Recon-Fabric-Pulnam-Microsoft-Catapult.pdf
-
[2] A Reconfigurable Fabric for Accelerating Large-Scale Datacenter Services, ISCA’14 https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/Catapult_ISCA_2014.pdf
-
[3] Microsoft Has a Whole New Kind of Computer Chip—and It’ll Change Everything
-
[4] A Cloud-Scale Acceleration Architecture, MICRO’16 https://www.microsoft.com/en-us/research/wp-content/uploads/2016/10/Cloud-Scale-Acceleration-Architecture.pdf
-
[5] ClickNP: Highly Flexible and High-performance Network Processing with Reconfigurable Hardware – Microsoft Research
-
[6] Daniel Firestone, SmartNIC: Accelerating Azure’s Network with. FPGAs on OCS servers.
Disclaimer: The materials in this article are sourced from the internet, and the copyright belongs to the original authors. If there are any copyright issues, please contact me for removal.
