🤖 Self-Evolution: How LLM Achieves Breakthroughs in General Domain Capabilities through “Three Kingdoms Kill”? — In-Depth Analysis of Multi-Agent Evolve (MAE)

🌟 Introduction: Breaking Through the “Data Drought” and “Reward Wall” in LLM Training

Large Language Models (LLMs) have shown tremendous potential in reasoning tasks, but their capability enhancement is often constrained by two core challenges: extreme reliance on high-quality human-annotated data and difficulties in obtaining verifiable reward signals. Particularly when models aim to surpass human intelligence and venture into general open domains (such as common knowledge and complex reasoning), existing methods, including self-play mechanisms that rely on external environments (like code interpreters), seem inadequate.

So, is there a method that allows LLMs to break free from human “supervision” and achieve self-learning and evolution in general domains without true value answers?

A research team from the University of Illinois at Urbana-Champaign, Peking University, and NVIDIA has proposed a revolutionary framework: Multi-Agent Evolve (MAE), which enables LLMs to achieve unsupervised self-improvement through multi-agent collaborative evolution.

1. 🔍 Research Problems and Challenges

Core Problem: How to construct an effective Reinforcement Learning (RL) framework that allows LLMs to achieve self-improvement in general domains without human annotation.

Challenges Faced:

• • Limitations of Data and Rewards: Traditional LLM reinforcement learning heavily relies on expensive human-curated datasets and verifiable rewards (i.e., a standard answer is required for scoring), which limits the scalability and generality of model training.
• • Domain Limitations of Self-Play: Existing self-play methods primarily succeed in “closed environments” with clear feedback, such as Go or scenarios with code interpreters.
• • The Dilemma of General Domain Reward Signals: In open domains like natural language reasoning and general knowledge, reward signals are inherently ambiguous and difficult to quantify, making it crucial to design reward mechanisms that effectively enhance LLM general capabilities.

2. 🧑💻 Who is the Research Team?

The paper is co-authored by scholars from the University of Illinois at Urbana-Champaign, Peking University, and NVIDIA.

• • Main Authors: Haofei Yu, Tao Feng, Yixing Chen, Yiding Wang, Siqi Zhu, Muhan Zhang, Mostofa Patwary, Jiaxuan You.
• • Project Address: https://github.com/ulab-uiuc/Multi-agent-Evolve.

3. 🧠 What Method is Proposed? — The “Three-Body” Collaborative Evolution of LLM

3.1 Overview of the Method: Multi-Agent Evolve (MAE)

MAE is a multi-agent self-evolution framework whose core design involves instantiating a single LLM into three interacting roles (agents):

1. 1. Proposer: Responsible for generating new, challenging questions.
2. 2. Solver: Responsible for attempting to answer these questions.
3. 3. Judge: Responsible for evaluating the quality of the Proposer’s questions and the correctness of the Solver’s answers, providing reward signals.

These three roles form a closed-loop “question-solution-judgment” pipeline, synchronously optimizing their behaviors through reinforcement learning, enabling LLMs to self-evaluate and self-improve without external supervision.

3.2 Example Illustration: MAE’s “Three Kingdoms” Cooperation and Competition

The MAE framework cleverly designs a “domain-independent self-reward mechanism” that places the Proposer and Solver in a relationship of both competition and collaboration, driving the model to continuously evolve.

Agent (Role)	Main Task	Core Goal	Reward Mechanism (Key to Driving Evolution)
Proposer	Generate new questions	Propose high-quality and challenging questions for the current Solver.	1. Quality Reward: Judge evaluates the clarity and solvability of the questions. 2. Difficulty Reward: Increases when the Solver fails. The difficulty reward calculation formula is , meaning the lower the average score of the Solver, the higher the reward.
Solver	Answer the Proposer’s questions	Generate accurate and well-reasoned answers	1. Judge Reward: Judge scores directly based on the quality and correctness of the answers.
Judge	Evaluate questions and answers	As a generative reward model, provide numerical scores to guide the training of Proposer and Solver	1. Format Reward: Incentivizes Judge to output structured, interpretable formats (e.g., <span><score>X</score></span> tags).

The key “competitive” mechanism is that the Proposer is rewarded for presenting difficult problems, while the Solver is rewarded for providing correct answers. This mechanism drives the Proposer to continuously explore the boundaries of LLM capabilities and ensures that the Solver keeps learning to tackle increasingly difficult tasks, thus achieving collaborative evolution.

4. ✅ How to Validate the Effectiveness of the Method?

The team used Qwen2.5-3B-Instruct as the base model for experiments.

• • Outstanding Performance Improvement: MAE achieved an average improvement of 4.54% across multiple benchmark tests in mathematics, coding, reasoning, and general knowledge.
• • Surpassing Baselines and SFT:
• • Even in minimal resource settings (zero-shot initiation, using only 16 model-generated questions to guide the domain), MAE’s performance has surpassed strong baseline model AZR.
• • When initiated with a small number of unlabeled reference questions (without using true value answers), MAE’s performance is significantly better than that of standard supervised fine-tuning (SFT) baselines that used true value answers for training on the same dataset.
• • Effectiveness of Core Mechanism: Research indicates that generating feasible yet challenging questions is key to performance improvement. This difficulty-aware reward aligns with the psychological concept of the “Desirable Difficulty Effect”, which suggests that tasks need to be challenging but not overwhelming to achieve optimal long-term learning.

5. ⚠️ The “Ceiling” of the Evolutionary Path — Analysis of MAE Framework Limitations

The breakthrough nature of the MAE framework is undeniable, but any self-learning system has inherent limitations. A deep analysis of these constraints helps us understand the future direction of agent design.

5.1 Level One: The “Cognitive Ceiling” of Data and Judgment

This is the most direct and fundamental limitation, relating to the upper limit of intelligence that the system can achieve.

1. The Evolution Trap of “Garbage In, Garbage Out” (Local Optima)

The fuel for the evolution of the MAE system is self-generated data. If the initial model has capability and knowledge blind spots, then the questions and answers it generates will also carry the same blind spots. This may lead to evolution stagnating at a “local optimum”, for example, the model may delve deeper into a certain type of mathematical problem but fail to actively discover and explore areas where it is not proficient, such as philosophical reasoning or creative writing. The system finds it difficult to break through its initial cognitive boundaries.

2. The Judge’s Ability is the Ultimate Bottleneck of the System

The entire system’s evolutionary direction is entirely guided by the Judge role. If the Judge’s own ability is limited and cannot recognize higher-level, more nuanced correctness, logic, or creativity (for example, it can only judge basic logic but cannot appreciate advanced artistic expression), then it cannot guide the Proposer and Solver towards higher goals. This means that the intelligence upper limit of the system cannot exceed the judgment ability of its Judge.

5.2 Level Two: The “Conductor’s Dilemma” in Reward Design

Reward signals are the “conductors” that guide agent behavior, and their design directly determines the quality and breadth of evolution.

1. The Subtle Balance of Difficulty and Quality: Beware of “Bad Problems”

The core of MAE is to reward the Proposer for presenting “difficult problems”. This is a double-edged sword: if the difficulty reward is not strongly counterbalanced by the quality reward, the Proposer may tend to generate unsolvable or ambiguous “bad problems” (e.g., logical paradoxes, insufficient information problems, pure word games) to “trick” the system into obtaining high scores. While this behavior satisfies the reward function, it does not help the Solver’s capability enhancement. Therefore, designing a robust problem quality filtering mechanism is crucial.

2. The Dilemma of Exploration vs. Exploitation

In reinforcement learning, agents need to balance between exploring new domains (which may yield high returns) and exploiting known domains (which provide stable returns). The MAE system may, in order to stably obtain rewards, remain in its areas of expertise, continuously generating and solving questions (e.g., only generating simple common-sense questions), while failing to explore entirely new, unfamiliar domains (such as interdisciplinary reasoning or complex programming). This tendency towards “comfort” limits the breadth of its general capabilities.

5.3 Other Limitations

• • Stability of Training: The paper analysis mentions the need to focus on stability issues such as dataset corruption. To address this, MAE integrates multiple safety mechanisms such as format rewards and quality filtering.
• • Scale and Unification: The research team plans to expand to larger-scale base models and integrate more roles and verifiable environments to build a unified self-evolution platform, indicating further exploration space for the current framework in terms of scale and generality.

6. 💡 Insights for Agent Architecture Design

The MAE framework provides profound insights for the design of future LLM agent architectures:

1. 1. From “Monolithic” to “Evolvable Groups”: Agent architectures should not be viewed merely as single entities executing tasks but should be designed as self-evolving multi-role systems. By decomposing tasks into the three fundamental roles of generation, solving, and evaluation, an agent (or a group of agents) can form a closed-loop learning ecosystem.
2. 2. Designing “Internal Supervision” Rewards: Abandon reliance on external true value answers and instead design internal reward signals that push the agent’s own capability boundaries. Difficulty-aware rewards () are a paradigm that teaches agents to propose “beneficial challenges”, thus achieving efficient learning and exploration.
3. 3. Built-in Quality Control Mechanisms: Stability is the greatest challenge for agents generating self-produced data. Future agent systems must incorporate an **”internal reviewer” role** (like the Judge) and quality filtering mechanisms to ensure the quality of generated data, preventing “cheating” and data corruption, which is the cornerstone for the stable operation of agent frameworks.
4. 4. Role Specialization and Unified Models: MAE demonstrates that instantiating multiple specialized roles from a shared LLM backbone is feasible. This design achieves both specialization of role responsibilities (questioning, solving, judging) and maintains model unity and efficient training through synchronous parameter updates.

arxiv: https://arxiv.org/abs/2510.23595