Deep Learning in AI Papers: Multi-Layer Visual Feature Fusion in Multimodal LLMs: Methods, Analysis, and Insights

Author: Lin et al.PDF Link: https://arxiv.org/pdf/Lin_Multi-Layer_Visual_Feature_Fusion_in_Multimodal_LLMs_Methods_Analysis_and_CVPR_2025_paper.pdf

Introduction: Why Can We “See” Clearly but Not “Speak” Accurately?

Multimodal large language models (MLLMs) can now describe images, answer visual questions, and even perform OCR. But have you ever wondered:

• Which layers of visual features should be used for an image?
• At which position in the LLM should these features be “inserted” and in what manner?

This CVPR 2025 paper thoroughly dissects these two questions using over 50 experimental setups, 8 benchmarks, and 3 data scales. By the end, you will realize that having “multiple layers of visual features” is not necessarily better; the key lies in “how to select and how to fuse” them.

1. Research Background: The “Black Box” of Visual Encoders

Currently, mainstream MLLMs directly feed the last layer features of CLIP-ViT into the LLM, a simple yet potentially suboptimal approach.

• Controversy: Models like MiniCPM and LLaVA perform well with a single layer, while EVLM and Dense Connector show improved performance with multiple layers.
• Pain Point: How to select layers? Where to fuse? Should modules be added? All of this is based on empirical experience.

The authors abstract the problem into two main dimensions:

1. Visual Layer Selection (Which layers?)
2. Fusion Strategy (Where & How to fuse?)

Figure 1 illustrates four typical paradigms clearly:

• (a)(b) Single-layer vs Multi-layer Feature Extraction
• (c)-(f) External/Internal × Direct/Modular Four Fusion Strategies

2. Methodology: Two Criteria, Four Strategies

2.1 How to Select Visual Layers? — Two Paths: “Similarity” and “Proportion”

The authors propose two simple yet effective criteria:

1. Based on Similarity: Divide the 24 layers of ViT into three segments (beginning, middle, end) based on cosine similarity, selecting one representative from each segment.
2. Based on Proportion: Split into front, back, or all layers in a 1:1 depth ratio, aligning with existing work.

Experiments reveal:

• Selecting one layer from each stage (e.g., {3,18,23}) > selecting multiple layers from the same stage > selecting all layers.
• Too many layers can lead to performance drops, as LLMs struggle with redundant information.

2.2 How to Determine Fusion Strategy? — External vs Internal, Direct vs Modular

The authors break down fusion into a 2×2 matrix:

Position	Mode	Representative Implementation	Parameter Count	Characteristics
External (Input Side)	Direct	Dense Connector	Very Low	Stable, Easy to Scale
External	Modular	MMFuser	Medium	Learnable Weights
Internal (LLM Intermediate Layer)	Direct	DeepStack	Low	Requires LLM Structure Modification
Internal	Modular	Cross-Attention	High	Flexible but Hard to Train

Quick Overview of Formulas:

• Internal Fusion:
• External Fusion:

3. Experiments: Over 50 Ablation Studies, Conclusions Are Clear

3.1 Benchmarks and Setup

• Model: Mini-LLaVA (1.4B LLM + CLIP-ViT-L/14)
• Data: Pre-trained on 558k image-text pairs, instruction fine-tuned on 665k dialogues
• Evaluation: 8 tasks including GQA, MMBench, TextVQA, OCRBench, POPE, etc.

3.2 Internal Fusion Results

Modular (Cross-Attention)

• Double {3,18} average 48.74 > Single 48.60 > Triple 46.91
• All directly trained to failure (loss 6+ not converging)
• Early layers significantly aid OCR and detail tasks, while too many late layers can interfere.

Direct Fusion (DeepStack)

• Increasing the number of layers improves performance, indicating that direct addition is more favorable for LLMs.
• Latter outperforms Former, as LLMs have weaker attention on later tokens, resulting in smoother fusion.

3.3 External Fusion Results

• External Direct Fusion averages 49.88, the best overall, with the smallest variance.
• Modular (MMFuser) drops 0.69 when using All layers, indicating that complex modules are more sensitive to layer selection.

4. Further Analysis: Data Scale and Model Components

4.1 Larger Data, Better Internal Fusion

• As SFT data increases from 332k to 737k, the internal fusion curve steepens while external fusion saturates.
• Conclusion: With sufficient data, internal direct fusion can rival external methods.

4.2 Changing ViT and LLM Also Holds

• Replacing CLIP with SigLIP, external direct fusion improves from 49.18 to 51.76.
• Switching from a 1.4B LLM to a 2.7B model further enhances performance to 52.63.
• Internal modular performance drops to 43.27 under SigLIP, indicating a risk of overfitting with complex modules.

5. Practical Guide: How to Implement in Your Project?

Scenario	Recommended Configuration	Reason
Data < 1M	External + Direct Fusion	Stable, easy to train, no performance drop
Data > 1M	External or Internal + Direct Fusion	Releases internal potential with fewer parameters
Extremely Limited Resources	Select only {Beginning, Middle} two layers	Performance close to three layers, halving computation
Want to Rank High	External Direct + SigLIP + 2.7B LLM	Same SOTA combination as in the paper

6. Conclusion and Further Thoughts

This paper thoroughly explains the use of “two criteria + four strategies” for multi-layer visual features:

• Layer Selection: Select one representative from each stage, avoiding redundancy.
• Fusion: External direct is the most robust, internal direct has potential, while modular should be approached with caution.

Future considerations could include:

1. Dynamic Layer Selection: Can the model decide which layers to use?
2. Task Adaptation: Do OCR and VQA require different layer combinations?
3. End-to-End Optimization: Can we train the visual encoder together, breaking the “freeze ViT” limitation?

Feel free to like, comment, and share.