Introduction by Long: This paper presents a simple yet effective reinforcement learning framework, VL-DAC, which addresses the training difficulties of vision-language models in interactive tasks. Surprisingly, it achieves significant skill transfer with training conducted solely in inexpensive synthetic environments, providing a new approach for the practical training of AI agents.
Imagine you are playing a 3D maze game, but the character is controlled by an AI. It needs to decide whether to turn left, right, or go straight based on the visuals and text instructions it sees. This sounds simple, but it poses a significant challenge for AI! 🤔
VL-DAC training effect demonstration:
This paper from T-Tech proposes the VL-DAC framework, allowing AI to train in low-cost virtual environments and directly apply its skills to real-world scenarios. Remarkably, after training in simple environments like MiniWorld and ALFWorld, the AI’s performance in the BALROG game control task improved by 50%! 🎮VL-DAC architecture diagram:
Traditional methods either require meticulous parameter tuning or perform poorly in long-sequence tasks. VL-DAC achieves stable and efficient training by decoupling policy updates and value assessments. This means that in the future, we may train smarter AI assistants at a lower cost! 🤖
Original paper information:
Paper Title:
Enhancing Vision-Language Model Training with Reinforcement Learning in Synthetic Worlds for Real-World Success
Publication Date:
August 2025
Authors:
George Bredis, Stanislav Dereka, Viacheslav Sinii, Ruslan Rakhimov, Daniil Gavrilov
Published by:
T-Tech
Original Paper Link:
http://arxiv.org/pdf/2508.04280v1
Open Source Code Link:
https://github.com/corl-team/VL-DAC
Prospects of Reinforcement Learning in Synthetic Worlds
Synthetic worlds are becoming a new battleground for training AI agents. Traditional methods rely on real data collection, which is costly and inefficient. Synthetic environments like MiniWorld and ALFWorld can simulate various interactive scenarios at a very low cost. Although these environments are simple, they can cultivate basic skills in AI such as navigation and object manipulation, laying the groundwork for real-world applications.
Key Breakthrough: The research found that models trained in a single synthetic environment can significantly enhance performance in real-world scenarios. For example, models trained in ALFWorld achieved a 50% relative improvement in the BALROG game control task.
Interactive Challenges Faced by Vision-Language Models
Current vision-language models (VLMs) excel in static image descriptions but often struggle in interactive tasks. The main challenges include:
Long Sequence Decision Making: Requires continuous observation of visual inputs and making multi-step decisions.
Credit Assignment: How to accurately assign the final reward to each decision step.
Visual Dynamic Understanding: Requires real-time parsing of constantly changing visual scenes.
Analysis of Limitations in Existing Reinforcement Learning Algorithms
Current mainstream methods such as RL4VLM, LOOP, and ArCHer each have significant flaws:
RL4VLM: Relies on fragile hyperparameter λ, requiring careful tuning for each environment.
LOOP: Suffers from high noise in credit assignment for long-sequence tasks, easily falling into local optima.
ArCHer: Requires a large amount of offline data buffering and performs poorly in sparse reward scenarios.
Core Ideas and Innovations of the New Method VL-DAC
VL-DAC (Vision-Language Decoupled Actor-Critic) proposes an innovative decoupled architecture:
Token-Level Policy Updates: Applies PPO (Proximal Policy Optimization) updates to each action token.
Step-Level Value Assessment: The value function learns only at the environment step level, preventing gradient backflow to the main network.
This design eliminates the fragile weight terms of RL4VLM, avoids the sequence-level credit assignment issues of LOOP, and does not require the large replay buffer of ArCHer.
Experimental Design: Performance Comparison Across Multiple Environments
The research team validated the effectiveness of VL-DAC in several lightweight simulation environments:
MiniWorld: Tests navigation and path planning capabilities.
ALFWorld: Evaluates spatial reasoning abilities in household tasks.
WebShop: Tests long-term planning and web interaction capabilities.
Successful Cases from Synthetic Training to Real Tasks
Most surprisingly, models trained in inexpensive synthetic environments demonstrated strong transfer capabilities. Models trained in ALFWorld (a simulated household task environment) achieved a 50% relative improvement in the BALROG game control task, breaking the barrier between synthetic and real. This cross-domain transfer proves that the model acquires general decision-making abilities rather than specific environmental memory.
Table 2: Performance Comparison of BALROG under Different Prompting Strategies
In the WebShop (e-commerce browsing environment), a model trained for only 2k steps achieved a 2% absolute improvement in the VisualWebBench web interaction task. This “small investment, big return” characteristic makes VL-DAC a highly cost-effective training solution, akin to training a professional esports player using arcade machines in a game hall 🎮.
Analysis of VL-DAC’s Performance in Different Benchmark Tests
In the corridor navigation task of MiniWorld, VL-DAC’s success rate outperformed traditional methods. As shown in Figure 3, RL4VLM exhibited significant fluctuations at different λ values (highest 98%, lowest 55%), while VL-DAC consistently maintained above 93%. This stability stems from its unique dual-stream design:
Figure 3: Success Rate Comparison of VL-DAC and RL4VLM Across Six Environments
Token-Level Fine-Tuning
Applying PPO updates independently to each action token allows the model to fine-tune the angle value in “turn left 15” without affecting the action type.
Step-Level Evaluation
The value function learns at the environment step level, avoiding noise interference from token-level evaluations.
Figure 5: VL-DAC’s Continuous Improvement in Long Sequence Tasks While LOOP Converges Early
Experimental Validation of Stability and Scalability
Ablation experiments reveal the secrets of stability (Figure 4). As KL regularization, value warming, and gradient clipping are gradually added, the training curve transitions from severe oscillation to smoothness. These three techniques act like a stabilizing gimbal for reinforcement learning:
Figure 4: Changes in Training Curves After Gradually Adding Stability Techniques
KL Regularization: Prevents policy updates from deviating too far from the initial model, avoiding “amnesia.”
Value Warming: Allows the value function to learn to estimate returns before updating the policy.
Gradient Clipping: Prevents value assessment errors from contaminating the main network.
In terms of scalability, VL-DAC maintains stable performance on the 7B parameter Qwen2-VL and 4B Gemma models (Table 1). Traditional methods require re-tuning for different model sizes, while VL-DAC truly achieves “train once, use everywhere.”
Table 1: Stability Comparison of RL4VLM and VL-DAC Across Different Model Architectures
Future Research Directions: Exploring Environmental Scale and Complex Tasks
Although VL-DAC performs excellently in existing environments, there is still room for breakthroughs in three frontier directions:
Ultra-Long Range Planning: Current models perform poorly in tasks exceeding 100 steps and need to incorporate external memory modules.
Multi-Agent Collaboration: How to allocate credit in multi-agent environments remains an unsolved problem.
Physical Interaction: Expanding from discrete actions to continuous control (e.g., robotic manipulation).
The most promising direction is to build an open RL4VLM training ground—a collection of lightweight environments contributed by the community. Just like training a versatile game player with countless mini-games, ultimately combining them into a general intelligent agent. This “fragmented training” strategy aligns better with the principles of deep learning than creating a single complex environment.
Analysis of Experimental Results
The success of VL-DAC can be attributed to its decoupled design matching model characteristics. Vision-language models are inherently hierarchical: the lower levels process pixel features, the middle levels understand object relationships, and the higher levels make decisions. Traditional methods forcibly inject decision signals into all levels, leading to signal conflicts. VL-DAC’s value assessment only affects the high-level decision module, preserving the lower-level visual feature extraction capabilities.
In sparse reward scenarios (such as MiniWorld-WallGap), VL-DAC’s continuous improvement (Figure 5) exposes the fundamental flaw of LOOP: when the entire sequence has only one ultimate reward, LOOP assigns the same weight to each token, akin to throwing darts blindfolded. In contrast, VL-DAC’s step-level critic can identify key decision points, even when the final reward has not yet been obtained.
Experimental Results
Table 3 reveals the phenomenon of selective enhancement in skill transfer: training in ALFWorld primarily enhances route planning abilities (VSI-Bench +1.54%), while spatial directional perception remains largely unchanged. This supports the “environment determines skill” hypothesis—environments like ALFWorld that require navigation particularly reinforce spatial path optimization capabilities.
Table 3: Comparison of Results from Different Skill-Specific Tests
More encouragingly, reinforcement training did not harm foundational abilities (Table 4). On nine general benchmarks like MMBench, the trained models maintained their original levels or even showed slight improvements, breaking the “catastrophic forgetting” curse of reinforcement learning.
Table 4: Performance of Trained Models on General Benchmarks
Conclusion and Outlook on the Development of Vision-Language Models
VL-DAC demonstrates the feasibility of a low-cost reinforcement learning path: abandoning the pursuit of ultra-realistic simulators in favor of constructing diverse lightweight environments; replacing computational power accumulation with algorithmic innovation. This opens new possibilities for small and medium-sized teams to participate in the AGI competition—training intelligent agents with real-world skills using just 20 GPU hours instead of million-dollar computing clusters.
In the future, vision-language models will move towards closed-loop learning between synthetic and real: learning basic skills in synthetic environments, fine-tuning and adapting in real scenarios, and then feeding back experiences to synthetic environments. This “symbiotic model” may be the shortest path to achieving general intelligent agents.