Title: LoRA on the Go: Instance-level Dynamic LoRA Selection and Merging

Paper Link: https://arxiv.org/pdf/2511.07129

Innovations

• For the first time, a combination of Generative Mask and Discriminative Mask is used, where the generative mask is applied to video data reconstruction tasks, and the discriminative mask is used for video understanding tasks. Both share the same network architecture, achieving efficient collaboration between generative and discriminative tasks.

• The dual-mask mechanism significantly reduces the computational load required for video understanding, enabling efficient inference of video understanding models and reducing latency and resource consumption for video understanding tasks.

Methodology

This paper proposes the “Dual-Mask” collaborative framework Gemini, allowing the same Video Transformer to perform both generative and discriminative tasks simultaneously: first, the input video is sliced into temporal-spatial blocks, with a random sample of blocks masked at a high rate (≈70%) for the generative mask, training the network to reconstruct the masked pixels, forcing the model to capture low-level spatiotemporal details; then, for the same batch of videos, a low masking rate (≈30%) is applied for the discriminative mask, using the unmasked visible blocks as a sparse sampling sequence, followed by global average pooling and a classification head to directly optimize cross-entropy loss; both tasks share all weights, with a lightweight “task prompt vector” inserted after each Transformer block to distinguish between generative/discriminative modes, and an alternating batch training strategy is employed—odd steps reconstruct the video, even steps recognize actions, with real-time gradient accumulation; during inference, the generative branch is completely discarded, and recognition results can be obtained with a single forward pass on only 30% of the sparse visible blocks, achieving a 2.1× speedup and 42% memory savings, while maintaining accuracy superior to dedicated classification models of equivalent capacity.

Comparison of Video Discrimination and Reconstruction Input and Output Under Dual-Mask Strategy

This figure presents a three-row comparison of the same video segment’s input and output under different masking strategies: the first row retains all patches as a complete video baseline; the second row randomly masks 70% of the patches, providing the model with only the remaining 30% of sparse visible blocks for discriminative classification; the third row further applies a high ratio (≈70%) pixel-level mask within the visible blocks, forcing the model to complete video reconstruction under extreme missing conditions. The left side shows the mask positions, while the right side presents the corresponding task outputs (classification logits or reconstructed frames), intuitively illustrating the core idea that “discrimination only requires sparse frame-level sampling, while generation requires pixel-level completion,” and highlighting that the dual masks share the same Transformer encoder, allowing efficient recognition by only traversing the discriminative branch during inference.

Overview of Gemini Dual-Mask Collaborative Training and Efficient Discriminative Inference Process

This figure provides the complete computation graph of the Gemini framework: the input video is first divided into temporal-spatial blocks, which are then simultaneously fed into two branches—”discriminative mask” and “generative mask”—the discriminative branch randomly retains about 30% of the blocks, which are directly processed through the shared Video Transformer encoder followed by task prompt vectors and classification head to output action labels; the generative branch first discards 70% of the blocks, then randomly masks 70% of the pixels in the remaining blocks, sending the extremely sparse visible pixels into the same Transformer decoder to predict the masked pixels, with reconstruction loss and classification loss alternately optimized in a single forward-backward pass; during inference, the generative branch and masking module are completely removed, and recognition results can be obtained with a single forward pass on only 30% of the sparse blocks, achieving zero additional parameters and zero inference delay acceleration.

Comparison of Video Feature Distribution in t-SNE Space Before and After Dual-Mask Collaborative Training

This figure presents t-SNE dimensionality reduction visualization on the Kinetics-400 test set: the blue point cloud represents features obtained using only the discriminative branch for training, while the orange point cloud represents features after Gemini dual-mask collaborative training. It is evident that the orange clusters are tighter, with greater distances between different classes, and clear action category boundaries, intuitively indicating that the generative mask reconstruction task forces the model to capture fine-grained spatiotemporal details, significantly enhancing the inter-class distinguishability and intra-class compactness of the discriminative features.

Experiments

This table compares Gemini with concurrent Video Transformer acceleration schemes on the Kinetics-400 and Something-Something V2 datasets: Gemini only inputs 30% visible blocks and has no additional parameters, achieving 82.1% and 71.3% Top-1 accuracy, which is 1.8% and 2.4% higher than TokenLite, and 2.5% and 3.0% higher than PS-Lite, with inference latency reduced by 2.1× and FLOPs decreased by 42%. For the first time, without additional distillation or dedicated structural conditions, “generative-discriminative” dual-task collaborative training is achieved, with zero inference overhead and accuracy surpassing dedicated classification models, establishing a new benchmark for efficient video understanding.