🫱Click here to join the group chat for 16 subfields (🔥Recommended)🫲
Deepseek has open-sourced three code repositories today, focusing on “Optimizing Parallel Strategies”.
Among them, DualPipe is the most popular, and we found that the development team of this algorithm includes Liang Wenfeng himself.
The DualPipe dual pipeline parallel algorithm has already been used in training the Deepseek v3 model, and this innovative algorithm was mentioned in previous technical reports.
Specifically, the DualPipe algorithm achieves complete overlap of the forward and backward computation-communication phases while reducing pipeline bubbles.
To address the communication overhead caused by cross-node expert parallelism in the DeepSeek-V3 model, which leads to an inefficient computation-communication ratio maintained at about 1:1, the DeepSeek team designed this innovative pipeline parallel algorithm called DualPipe.
The core idea of DualPipe is to achieve overlap of computation and communication through paired forward and backward computation chunks.
Each computation chunk is divided into four components: attention computation, all-to-all dispatch, multi-layer perceptron (MLP), and all-to-all combine.
For the backward computation chunks, researchers specifically adopted the ZeroBubble method, further splitting the attention computation and MLP into two sub-modules for input backward propagation and weight backward propagation. Additionally, the system includes a pipeline parallel (PP) communication component.
As shown in the figure above, for paired forward and backward computation chunks, the execution order of these components is rearranged, and the resource allocation ratio of the GPU streaming multiprocessors (SMs) between communication and computation tasks is manually adjusted. Through this overlapping strategy, it ensures that all-to-all communication and pipeline communication are completely hidden during runtime. Based on this efficient overlapping mechanism, the complete DualPipe scheduling scheme is shown in the figure below.
The DualPipe scheme adopts a bidirectional pipeline scheduling technique, inputting micro-batches from both ends of the pipeline simultaneously, allowing most communication operations to be completely overlapped.This overlapping mechanism also ensures that when the model scale is further expanded, as long as a constant computation-communication ratio is maintained, fine-grained expert parallelism across nodes can still be achieved, while the overhead of all-to-all communication approaches zero.
Example of DualPipe:An 8-level pipeline parallel (PP) scenario with 20 micro-batches using the bidirectional DualPipe scheduling strategy. Due to the symmetry of micro-batches in backward propagation and forward propagation, we omit the batch ID annotations for simplicity. The two cells sharing a black border indicate mutual overlap of computation and communication.
In addition, even in general scenarios with light communication load, DualPipe still demonstrates efficiency advantages. The table above summarizes the pipeline bubbles and memory usage of different pipeline parallel (PP) methods.
As shown in the table, compared to ZB1P and 1F1B, DualPipe significantly reduces pipeline bubbles at the cost of only increasing peak activation memory by 1/PP times.
Although DualPipe requires maintaining two copies of model parameters, due to the use of a large expert parallel (EP) scale during training, memory consumption has not significantly increased. Compared to Chimera, DualPipe only requires that the number of pipeline stages and micro-batches be divisible by 2, without the need for the number of micro-batches to satisfy divisibility with the number of pipeline stages.
Furthermore, for DualPipe, both pipeline bubbles and activation memory do not increase with the number of micro-batches.
The following code is the core part of implementing DualPipe, achieving overlap of computation and communication by alternating processing of different phases (phase0 and phase1), improving GPU utilization.
def_forward_backward_compute_chunk(self, phase0:int, phase1:int)->None:
if self.forward_only:
self._forward_compute_chunk(phase0)
return
ifnot self.overlaped_forward_backward:
self._forward_compute_chunk(phase0)
self._backward_compute_chunk(phase1)
return
# pre-forward
phase0 ^= self.is_in_second_half
chunk_id0 = self.current_f_chunk_id[phase0]
self.current_f_chunk_id[phase0]+=1
module0 = self.module[phase0]
inputs0 = self.input_chunks[phase0][chunk_id0]
is_last_stage0 =(self.is_first_rank and phase0 ==1)or(self.is_last_rank and phase0 ==0)
if is_last_stage0 and self.criterion isnotNone:
labels0 = self.labels[phase0][chunk_id0]
criterion0 = self.criterion
else:
labels0 =[]
criterion0 =None
# pre-backward
phase1 ^= self.is_in_second_half
chunk_id1 = self.current_b_chunk_id[phase1]
self.current_b_chunk_id[phase1]+=1
module1 = self.module[phase1]
is_last_stage1 =(self.is_first_rank and phase1 ==1)or(self.is_last_rank and phase1 ==0)
if is_last_stage1:
loss1 = self.loss_chunks[chunk_id1]
outputs1 =[]
output_grads1 =[]
else:
loss1 =None
outputs1 = self.output_chunks[phase1][chunk_id1]
ifnot self.return_outputs:
self.output_chunks[phase1][chunk_id1]=None
output_grads1 = self.output_grad_chunks[phase1][chunk_id1]
self.output_grad_chunks[phase1][chunk_id1]=None
non_empty =[(t, g)for t, g inzip(outputs1, output_grads1)if g isnotNone]
outputs1, output_grads1 =list(zip(*non_empty))
# forward & backward
outputs0, loss0 =type(module0).overlaped_forward_backward(
module0, inputs0, criterion0, labels0,
module1, loss1, outputs1, output_grads1,
)
# post-forward
if(not is_last_stage0)or self.return_outputs:
self.output_chunks[phase0].append(outputs0)
if is_last_stage0 and self.criterion isnotNone:
self.loss_chunks.append(loss0)
# post-backward
inputs = self.input_chunks[phase1][chunk_id1]
self.input_chunks[phase1][chunk_id1]=None
input_grads1 =[t.grad for t in inputs]
self.input_grad_chunks[phase1].append(input_grads1)
Steps for phase0 and phase1 are as follows:pre-forward phase 01. phase1 ^= self.is_in_second_half switches the phase for the second half using XOR operation2. Get input data from input_chunks and select the model module in phase0.3. If the current phase is the last phase (is_last_stage0), prepare labels and loss function for loss calculation.pre-forward phase 11. Similar to the forward phase, phase1 is switched using XOR.2. Get output gradients from output_grad_chunks, filter invalid gradients, and prepare data for backward propagation.3. If it is the last phase, get loss values from loss_chunks; otherwise, prepare output gradients for chain rule differentiation.forward & backward1. Call the overlaped_forward_backward method to perform forward propagation (phase0) and backward propagation (phase1) simultaneously. Pass the forward outputs and loss to the backward phase, utilizing asynchronous computation and CUDA streams to achieve computation overlap, reducing bubble time.post-processing1. If not the last phase or outputs need to be retained, store the forward results in output_chunks; save the loss value in the last phase.2. After backward propagation, extract input gradients (input_grads1) and store them for parameter updates.Project addresses:DualPipe:https://github.com/deepseek-ai/DualPipeExpert Parallelism Load Balancer:https://github.com/deepseek-ai/eplbProfiling Data:https://github.com/deepseek-ai/profile-dataWhat do you think, on the last day tomorrow, what will DeepSeek open source?[Deep Blue AI]will continue to share cutting-edge dynamics in the field of artificial intelligence👇～[Deep Blue Academy] Large Model Group Chat～💪Deep Blue Academy is committed to helping friends “break the invisible wall“, co-building a more free, deeper, and valuable communication ecosystem community!Those working in major enterprises can refer each other; those pursuing graduate studies can exchange ideas and collaborate.🚀 Welcome to join the exclusive circle of thousands of large model research/engineering peers:
· Insight into the frontier: Get the “Large Model Technology Weekly” every week (including paper interpretations/technical solutions/industry dynamics).
Scan to add Alang and enter the large model group chat
(Invitations are based on submission order, please join as soon as possible)
👇
Group chat seen👋🏻
Past ReviewJust now! Detailed explanation of DeepSeek’s major open source: only 300 lines of code, extremely simple and powerful512 drones! How does GCBF+ achieve large-scale multi-agent dynamic obstacle avoidance?