NCCL

NCCL Communication C++ Example (Part 4): Implementation and Analysis of AlltoAll_Split

by
NCCL Communication C++ Example (Part 4): Implementation and Analysis of AlltoAll_Split

In NCCL cluster communication operations, there is no direct alltoall communication API, nor is there an alltoall_split method. This article mainly introduces the principle of alltoall_split and one implementation method, along with profiling of the send/recv operator calls in the alltoall operation. Other references: NCCL Communication C++ Example (Part 1): Basic Use Case Interpretation and … Read more

NCCL Communication C++ Example (Part 2): Establishing Multi-Machine Connections Using Sockets

by
NCCL Communication C++ Example (Part 2): Establishing Multi-Machine Connections Using Sockets

Click the blue text to follow us The NCCL (NVIDIA Collective Communication Library) is primarily used for GPU cluster communication. This article provides some basic C++ API library calls for learning reference. It mainly introduces how to establish multi-machine initialization connections using TCP/IP sockets.. The multi-machine example on the NCCL official website uses MPI to … Read more

NCCL Communication C++ Example (Part 1): Basic Use Case Interpretation and Execution

by
NCCL Communication C++ Example (Part 1): Basic Use Case Interpretation and Execution

Click the blue text to follow us The NCCL (NVIDIA Collective Communication Library, pronounced “Nickel”) library is primarily used for communication in GPU clusters. This article provides some basic C++ API library calls for learning reference. The NCCL documentation has a rich API introduction, but the explanations are interspersed with various operational details, which can … Read more

Understanding Nvidia’s Multi-GPU Communication Framework NCCL

by
Understanding Nvidia's Multi-GPU Communication Framework NCCL

According to Lei Feng Network, this article is based on Tan Xu’s answer to the question “How to Understand Nvidia’s Multi-GPU Communication Framework NCCL?” on Zhihu, and Lei Feng Network has obtained authorization for reprint. Question Details: In deep learning, multi-GPU parallel training is often required, and Nvidia’s NCCL library NVIDIA/nccl (https://github.com/NVIDIA/nccl) is frequently used … Read more