RTR Team Submission
Intelligent Gorilla Robot Compilation
Author Introduction:The co-first authors of this article are Hu Kaizhe, a PhD student from Tsinghua University’s Institute for Interdisciplinary Information Research, and PhD students Shi Haochen from Stanford University, C. Karen Liu, and Shuran Song.The article was completed during Hu Kaizhe’s visit to Stanford University. The corresponding authors are Professor C. Karen Liu from Stanford University’s Computer Science Department and Assistant Professor Shuran Song from Stanford University’s Electrical Engineering Department.
The motion control of humanoid robots is becoming the next hot research area for the application of Reinforcement Learning (RL) algorithms. Currently, mainstream solutions mostly follow the paradigm of “Sim-to-Real“. Researchers train general control models in thousands of simulation environments with different physical parameters through Domain Randomization, hoping that it can directly adapt to the real world with unknown dynamics due to its strong generalization ability. Although such “Zero-Shot Transfer“ solutions have achieved excellent performance in various motion tasks, their essential goal is to train a conservative policy that can be used in any environment. This strategy sacrifices the performance ceiling of the robot in specific real environments, as the real-world performance is the only important criterion for final deployment.
To break through this bottleneck, recent works have begun to explore fine-tuning models with a small amount of real data after simulation pre-training. For example, researchers from institutions such as NVIDIA and CMU proposed ASAP , which quickly compensates for the dynamic differences between simulation and reality by training a residual network; while the classic academic RMA (Rapid Motor Adaptation) algorithm has also been applied to bipedal robots, inferring environmental dynamics information from historical actions through an adaptation module. However, most of these works still focus on one-time compensation for dynamic deviations, and the interaction paradigm is more inclined towards offline learning, without continuously adjusting the model itself in real environments. Due to the extreme instability of humanoid robots, any small mistake can lead to expensive hardware damage, thus direct reinforcement learning training in real environments has long been regarded as an insurmountable obstacle.
Drawing inspiration from how human parents teach infants to walk, the Stanford team creatively proposed the soft-hardware collaborative system RTR (Robot-Trains-Robot).This system uses a “teacher“ robotic arm to guide the “student“ humanoid robot in online reinforcement learning in the real world. The paper related to RTR has been included in the CoRL 2025 conference, and the project code has been fully open-sourced.
RTR system highlights the importance of active physical assistance provided by robot teachers for achieving real-world reinforcement learning in humanoid robots. At the same time, to address the high cost of real-world data collection, the team also proposed a novel reinforcement learning algorithm that quickly adjusts the robot’s behavior by optimizing only a low-dimensional latent variable related to environmental dynamics, greatly improving sample efficiency. This contribution in algorithms further liberates the potential of real-world reinforcement learning, significantly surpassing traditional online system identification benchmarks such as RMA .
-
Paper Title:Robot Trains Robot: Automatic Real-World Policy Adaptation and Learning for Humanoids
-
Paper Link:https://arxiv.org/abs/2508.12252
-
Project Address:
https://robot-trains-robot.github.io/
-
Open Source Code: https://github.com/hukz18/Robot-Trains-Robot
-
Inclusion Status:CoRL 2025
1
Method
The RTR system consists of hardware and algorithm components.
1.1Hardware Setup: Collaborative Ecosystem of Teacher and Student
The hardware system of RTR consists of two groups of robots: “teacher“ and “student“.
Teacher System: The core is a UR5 six-axis robotic arm equipped with force/torque sensors. Four elastic cables connect the end of the robotic arm to the shoulder of the humanoid robot, allowing smooth transmission of assistive forces and avoiding impacts caused by rigid connections. In walking tasks, a programmable treadmill serves as an aid, ensuring that the student remains within the teacher’s arm span. A mini PC acts as the “brain” of the teacher system, processing force feedback signals and controlling the behaviors of the robotic arm and treadmill.
During the process of using a “teacher“ robotic arm to guide the “student“ humanoid robot in online reinforcement learning, the teacher robotic arm plays multiple key roles. It serves as a protective safety “harness“ to prevent the student from falling; it is also an automatically resetting“helper“, which can quickly lift the student after a failure to continue training; it is also a sensitive “signal source“, collecting valuable training data through force sensors to provide the student with reward signals that are difficult to obtain in real environments; and it is a wise “coach“, helping the student quickly adapt to the environment by setting curriculum learning progress and applying adversarial disturbances, and gradually increasing training difficulty to enhance the student’s learning efficiency and strategy robustness.
Student System: The student is based on the open-source ToddlerBot humanoid robot, which has 30 degrees of freedom, a compact size, and robust hardware, suitable for long-term continuous training. The open-source robot design makes it easier to modify as needed, and a separate workstation is responsible for running the reinforcement learning algorithm while continuously sending the latest policy model to the student for execution based on the student’s mobile data.
1.2 Algorithm Design: Three-Stage EfficientSim-to-Real Fine-Tuning
To maximize data efficiency, RTR proposes a three-stage process for Sim-to-Real fine-tuning:
1. Simulation Training of Environment-Adaptive Policies. In a large number of domain-randomized simulation environments, a control policy is trained that accepts a latent variable z encoding the physical parameters of the environment (such as friction, damping, etc.) as input. This latent variable is integrated into the policy network through a FiLM (Feature-wise Linear Modulation) layer, allowing the robot’s policy to adaptively adjust according to different dynamic environments.
2. Optimizing General Initial Latent Variables. Since the physical parameters of the real world are unknown, directly deploying the model from the first stage faces the challenge of “how to set the latent variable z in the real environment. To address this, researchers reference the idea of domain randomization and first optimize a “general“ latent variable in simulation, ensuring it achieves good average performance across all encountered simulation environments. This can guide the policy to produce action outputs that can adapt to various environments, becoming a reliable starting point for real-world fine-tuning.
3. Real-World Online Fine-Tuning. The policy trained in simulation, along with the general latent variable, is deployed to the real robot as input to the environmental state. With the assistance of the teacher robotic arm, the policy network parameters are frozen, and only the low-dimensional latent variable is optimized online. This optimization method for the latent variable allows the robot’s gait to quickly adapt to the dynamic characteristics of the real world with minimal data, achieving extremely high fine-tuning efficiency and a stable fine-tuning process. In contrast, directly using reinforcement learning algorithms like PPO to fine-tune the policy network itself in the real world can easily lead to instability and dangerous actions.
2
Experiments
Researchers comprehensively validated the effectiveness of the RTR system through walking and “swinging“ tasks.
In the walking task of fine-tuning from simulation to reality, the robot accurately tracks the target speed on the treadmill. Ablation experiments demonstrate the superiority of the RTR system design:
Effect of Teacher Assistance: Compared to a fixed harness, the actively compliant “flexible“ robotic arm (XY Compliant) significantly enhances learning effectiveness. At the same time, transitioning from high assistance to zero assistance through a “curriculum learning“ strategy (Z Schedule) outperforms fixed strategies of either high or low assistance throughout.
Data Efficiency of Fine-Tuning Algorithms: The RTR proposed “fine-tuning latent variables“ method outperforms baseline methods that fine-tune the entire policy network or fine-tune residual networks in both data efficiency and final performance. After just 20 minutes of real-world training, RTR can double the speed of the walking policy pre-trained in simulation.
Researchers further designed comparative experiments to verify the effectiveness of adding latent variable conditional inputs about environmental dynamics to the policy network using FiLM layers, which outperformed directly concatenating environmental dynamic variables into the policy network’s observations. The real-world fine-tuning effects based on the proposed method of RTR also surpassed the online parameter identification baseline represented by RMA.
In addition to the Sim-to-Real tasks, a pure real-world reinforcement learning “swinging“ experiment was also designed to demonstrate the potential of the RTR system in direct reinforcement learning tasks. In this task, the humanoid robot needs to learn to coordinate its legs to swing like a child, maximizing the swing amplitude. The teacher robotic arm perceives the student’s swing phase through real-time force feedback, providing “pushes“ (assistance) or “damping“ (disturbance) at the appropriate time. Experimental results show that the active participation of the teacher in providing “help“ and “disturbances“ curriculum learning is more efficient than the teacher merely acting as a fixed harness, while the force information perceived by the teacher always serves as an important reward signal to guide the student’s training. The entire learning process started from scratch and learned a significantly periodic swinging action within 20 minutes.
3
Conclusion and Outlook
The introduction of the RTR framework not only provides a practical solution to the current bottleneck of humanoid robot deployment and training but, more importantly, it introduces a new paradigm of active force assistance to aid the real-world learning of robots. This highly scalable framework can be extended in the future by using industrial robotic arms with greater load capacity or gantry systems with force sensors to promote the successful experience of RTR to full-sized humanoid robots and other complex robotic systems in real-world reinforcement learning tasks.
END
Intelligent Gorilla Matrix has various experts, click the business card to follow