According to Lei Feng Network, this article is based on Tan Xu’s answer to the question “How to Understand Nvidia’s Multi-GPU Communication Framework NCCL?” on Zhihu, and Lei Feng Network has obtained authorization for reprint.
Question Details:
In deep learning, multi-GPU parallel training is often required, and Nvidia’s NCCL library NVIDIA/nccl (https://github.com/NVIDIA/nccl) is frequently used in multi-GPU parallelism across major deep learning frameworks (Caffe/Tensorflow/Torch/Theano). How can we understand the principles and characteristics of NCCL?
Answer:
NCCL stands for Nvidia Collective multi-GPU Communication Library, which is a library that implements collective communication (all-gather, reduce, broadcast) for multiple GPUs. Nvidia has made many optimizations to achieve high communication speeds over PCIe, NVlink, and InfiniBand.
Below, I will introduce the features of NCCL from the following aspects: basic communication primitives, ring-based collectives, NCCL implementation on single-machine multi-GPU and multi-machine multi-GPU, and finally share some practical experiences using NCCL.
(1) Communication Primitive
Communication for parallel tasks can generally be divided into Point-to-point communication and Collective communication. P2P communication involves only one sender and one receiver, making it relatively simple to implement. The second type, Collective communication, involves multiple senders and multiple receivers. Common communication primitives include broadcast, gather, all-gather, scatter, reduce, all-reduce, reduce-scatter, and all-to-all. Here are a few commonly used operations:
Reduce: Receives data from multiple senders and combines it at one node.
All-reduce: Receives data from multiple senders and combines it at each node.
Traditional Collective communication assumes that the topology of the communication nodes forms a fat tree, as shown in the figure below, which maximizes communication efficiency. However, the actual communication topology can be quite complex and may not resemble a fat tree. Therefore, ring-based Collective communication is generally used.
(2) Ring-based Collectives
Ring-based collectives connect all communication nodes in a unidirectional ring, where data is transmitted sequentially around the ring. For example, suppose there are four GPUs, with GPU0 as the sender sending information to the remaining GPUs, transmitted in a ring manner: GPU0 –> GPU1 –> GPU2 –> GPU3. If the data size is N and the bandwidth is B, the total transmission time is (K-1)N/B. The time increases linearly with the number of nodes, which is not very efficient.
Next, the data to be transmitted is divided into S parts, with only N/S data transmitted each time, as shown in the following transmission process:
After GPU1 receives a portion of the data from GPU0, it then passes it to the next node in the ring, and so on. The total time taken is S*(N/S/B) + (k-2)*(N/S/B) = N(S+K-2)/(SB) –> N/B, provided that S is much greater than K, meaning the number of data parts is greater than the number of nodes, which is easily satisfied. Therefore, communication time does not increase with the number of nodes; it only depends on the total amount of data and bandwidth. Other communication operations like reduce and gather follow similarly.
So, in a scenario where GPUs are the communication nodes, how is the communication ring constructed? As shown in the figure below:
Single machine with 4 cards connected through the same PCIe switch to a CPU:
Single machine with 8 cards connected through different PCIe switches under two CPUs:
(3) NCCL Implementation
NCCL is implemented as CUDA C++ kernels and includes three primitive operations: Copy, Reduce, and ReduceAndCopy. Currently, NCCL version 1.0 only supports single-machine multi-GPU communication through PCIe, NVlink, and GPU Direct P2P. NCCL 2.0 will support multi-machine multi-GPU communication, with inter-machine communication via Sockets (Ethernet) or InfiniBand with GPU Direct RDMA.
The figure below shows that multi-GPU communication within a single machine occurs through PCIe and CPU sockets, while inter-machine communication occurs via InfiniBand.
Similarly, within a multi-machine multi-GPU setup, a communication ring must also be formed.
Below is a bandwidth speed variation chart for various operations on a single machine with 4 cards (Maxwel GPU) as communication volume increases. The bandwidth limit can reach 10GB/s, close to PCIe bandwidth.
The following image compares the speed of Allreduce under different architectures on a single machine:
Without considering the DGX-1 architecture, which is Nvidia’s deep learning platform capable of reaching 60GB/s bandwidth, the first three are typical single-machine multi-GPU connection methods. The third method has all four cards connected to a single PCIe switch, achieving a bandwidth of >10GB/s. The second method connects two GPUs through a switch and then through a CPU, resulting in slightly lower speed, while the first method connects two GPUs through a CPU and then through QPI to two cards on another CPU, resulting in the slowest speed, but still achieving >5GB/s.
The following image shows the performance of Allreduce under multi-machine conditions. The left image shows two machines with 8 cards, achieving >10GB/s speed with PCIe within the machine and InfiniBand between machines, which can essentially reach the communication speed of intra-machine.
The following image shows the scalability of NCCL on CNTK ResNet50, where 32 cards can achieve a nearly linear speedup.
(4) Our Practical Experience
First, we tested the GPU connection topology on a machine with a K40 GPU, as shown below:
It can be seen that the first four cards and the last four cards are connected through different CPU groups. GPU0 and GPU1 are directly connected through a PCIe switch, and then connected to GPU2 and GPU3 through the CPU.
The following shows the bandwidth test of PCIe. It can be seen that the communication between GPU0 and GPU1 can reach 10.59GB/s, while communication between GPU0 and GPU2~3 is slower due to passing through the CPU, and communication with GPU4~7 is even slower due to passing through QPI, but can still reach 9.15GB/s.
Through NVlink, the communication speed between GPUs can reach 35GB/s:
NCCL performs differently across various deep learning frameworks (CNTK/Tensorflow/Torch/Theano/Caffe) due to differing model sizes and batch sizes. For instance, in CNTK, Resnet50 can achieve linear speedup with 32 cards, whereas in NMT tasks, the speedup may not be as significant. This is because factors affecting parallel computing efficiency primarily include the number of parallel tasks, the workload of each task, and communication time. We must consider not only the absolute communication volume but also whether communication and computation can occur simultaneously and the computation/communication ratio. If the communication time is a smaller proportion of the computation time, then parallel computing tasks will be more efficient. NMT models are generally larger, often tens or hundreds of megabytes, unlike current image models which can be a few megabytes in size, resulting in a higher proportion of communication time.
The following is a simple comparison of single-machine multi-GPU acceleration for NMT models:
Above is some understanding of NCCL. Much of the material is also derived from NCCL’s official documentation. Welcome to exchange and discuss.
Intelligent Investment Advisory Advanced Training Class
With the gradual popularization of public internet financial concepts, the scale of financial management has expanded accordingly. The emerging intelligent investment advisory services, which are low-cost, risk-diversified, and emotionless, are increasingly accepted by the middle class and affluent classes. Dr. Wang Zhen will guide you on the path of intelligent investment advisory with real projects. For details, click to read the original link or long press to identify the QR code below~
