Multi-Agent Debate Tree ToD: Empowering AI With Critical Thinking

Click the blue text above to follow me

This article: 7200 words, 18 minutes reading

As AI tools become increasingly popular, tools like Deep Research are becoming more user-friendly, leading to an explosive growth in scientific research outputs. For example, arXiv received over 24,000 paper submissions in October 2024 alone. This astronomical growth presents unprecedented challenges for researchers: how to accurately identify new findings among a vast sea of research results? How to discern subtle differences between related papers? This issue becomes even more complex when different research communities propose similar ideas from different perspectives.

Traditional literature analysis methods often rely on simple text comparison or keyword matching, which struggle to capture the deep connections and differences between papers. Even the most advanced large language models often fall into superficial summaries during deep literature analysis, lacking true critical thinking abilities. Researchers need an intelligent system that can simulate human experts in academic discussions, one that not only understands the content of papers but can also conduct in-depth comparative analysis and critical thinking.

Even more disheartening is the fact that many LLMs currently engage in fabricating literature, fake citations, and other distressing practices. This compels us to discover new methods and tools to better harness AI.

This article will detail a groundbreaking framework—Tree-of-Debate. This framework achieves deep analysis and comparison of scientific literature through an innovative Multi-Agent architecture. Unlike traditional text analysis methods, the debate tree simulates the academic discussion process between human experts, stimulating true critical thinking through structured debates.

This research was completed by a research team from the Department of Computer Science at the University of Illinois, Urbana-Champaign. The team has a strong research background in natural language processing and artificial intelligence.

Image created by Xiumao

I wrote a Multi-Agent based on the methods of the paper and conducted a debate practice using this paper and a similar paper from the end of last year. Below are screenshots of the operation.

Swipe up and down to see more

Slide left and right to see more

Another paper titled “GENERATIVE ADVERSARIAL REVIEWS: WHEN LLMS BECOME THE CRITIC” (生成对抗评论：何时LLMs成为批评家) is available for download with Arxiv ID: 2412.10415v1 if you are interested.

Achieving Critical Thinking Through Debate

How the Debate Tree Works

Let’s understand how the debate tree works through a specific example. Figure 1 shows the debate structure constructed by the system while analyzing two papers on the reasoning capabilities of large language models (LLMs). This tree structure is not pre-defined but is dynamically generated during the debate process.

The above figure shows the core structure of the debate tree framework and a practical application case. In this example, the system analyzes two papers on LLM reasoning capabilities. The debate tree starts from the root node “LLM reasoning” and branches out into three main discussion topics:

1. Backtracking: exploring the error correction capabilities of LLMs in the reasoning process
2. Reasoning Evaluation: discussing the pros and cons of different evaluation methods
3. Multiple Paths: analyzing the effects of parallel reasoning paths

This structure showcases a core advantage of the debate tree: it can automatically identify and organize discussion topics, forming a hierarchical analysis framework. For instance, under the reasoning evaluation branch, the system further elaborated on three sub-topics: automatic evaluation, domain expert evaluation, and general human evaluation. Through this detailed breakdown, the system identified a key difference: Paper A did not use domain expert evaluation for complex reasoning tasks, while Paper B involved five domain experts in the evaluation of chemical reasoning. Such detailed findings can only be revealed through multiple rounds of debate and in-depth questioning.

The Dynamic Debate Process

Figure 2 illustrates the dynamic operation process of the debate tree. Each debate session consists of three key stages: Self-Deliberation, Topic Generation and Expansion, and Structured Debate. This process is carried out by multiple intelligent Agents, each with specific roles and responsibilities.

In the self-deliberation stage, the Paper Agent deeply reflects on the paper it represents. They not only extract relevant excerpts from the paper but also identify core innovations and anticipate potential challenges. This self-deliberation mechanism simulates the preparation process of human experts before a debate, ensuring that subsequent discussions are deeper and more targeted.

During the topic generation stage, the Moderator Agent determines the most valuable discussion topics based on the results of self-deliberation. These topics are not pre-defined but are dynamically generated based on the paper content and the progress of the debate. For instance, in Figure 2, we can see how the system gradually unfolds multiple related discussion branches from an initial topic, each representing an aspect worth exploring in depth.

In the structured debate stage, the system adopts a three-round dialogue model called “Present-Respond-Revise.” This model allows the Paper Agent to continuously deepen and refine its arguments while also posing strong challenges to the opposing viewpoint. The entire process is monitored and guided by the Moderator Agent, ensuring the debate remains on the right track.

Looking at Specific Cases

The paper introduces the specific discussion content of the Tree of Thoughts (ToT) and Chain of Thought (CoT) methods in the three rounds of debate:

• Present Round: each stating core advantages
• Respond Round: mutual questioning and challenges
• Revise Round: refining and adjusting arguments

Let’s look at a practical debate case (as shown in Table 3) on the topic of “The Generalization and Flexibility of Prompting Methods.” In this debate, the Tree of Thoughts (ToT) and Chain of Thought (CoT) methods engaged in in-depth discussions. Through three rounds of structured dialogue, the system successfully revealed the strengths and weaknesses of these two methods in different aspects.

In the first round (Present), ToT emphasized its multi-path exploration and backtracking abilities, arguing that this enables it to handle more complex reasoning tasks. CoT, on the other hand, highlighted its simplicity and good generalization capabilities, claiming that simple chains of thought are more adaptable to different types of problems.

In the second round (Respond), both sides began to question each other. ToT pointed out that simple linear thought chains might fail to capture the full complexity of the problem when dealing with complex reasoning tasks. CoT countered that overly complex reasoning mechanisms could impact the model’s practicality and generalizability.

In the final revision round (Revise), both sides refined their arguments. ToT clarified its advantages in complex reasoning tasks, acknowledging that it might seem overly complex for simpler tasks. CoT emphasized its success in balancing simplicity and effectiveness while also admitting that more complex reasoning mechanisms might be needed for certain complex tasks.

Experimental Validation: The Effectiveness of the Debate Tree

The table above shows the performance of the Tree-of-Debate system on different evaluation metrics:

• Breadth: improved by 6.85%
• Context: improved by 25.98%
• Factuality: maintained at a high level

The experimental results of the researchers (as shown in Table 2) demonstrate the superiority of the debate tree method. In all comparative scenarios, this method significantly outperformed baseline methods. Specifically, it improved breadth by 6.85%, meaning the system can more comprehensively cover important points in the papers. The improvement in context was even more significant, reaching 25.98%, indicating that the system could better understand and present the logical relationships between arguments. Meanwhile, the system maintained a high level of factual accuracy, ensuring that the generated analysis results are reliable.

Notably, even when there were no direct citation relationships between the papers, the system was still able to conduct effective comparative analysis. This proves the powerful ability of the debate tree in handling complex literature comparison tasks. Through ablation experiments, researchers also found that the tree structure and self-deliberation mechanism are key components of the system’s performance. Removing any one component would lead to a significant decline in analysis quality.

My Experimental Setup and System Implementation

To validate the effectiveness of the debate tree framework, I implemented a relatively complete prototype system (as shown in the scrolling image at the beginning). It took me most of a day to discover this research and complete the experiment. There may still be issues with methodological rigor and insufficient data. However, the original authors did not release the code, so there remains room for improvement. I will share the code in the Agent group in three days. The system uses the following tech stack:

1. Reasoning Engine:

• Core Model: Volcano Engine (based on Deepseek-R1)
• Context Window: 8K tokens
• Temperature Parameter: 0.3 (to ensure output stability)
• Sampling Strategy: Top-p = 0.9

Semantic Analysis:

• Vector Model: Jina Embeddings V3
• Vector Dimension: 768
• Similarity Threshold: 0.8
• Batch Size: 16

System Architecture:

• Memory Capacity: 1000
• Top-k Retrieval: 5

• Maximum Node Count: 1000
• Edge Weight Threshold: 0.5

• Evidence Support Threshold: 0.6
• Maximum Argument Count: 3
• Evidence Pool Capacity: 100

• Topic Generation Threshold: 0.7
• Maximum Rounds: 5
• Dynamic Expansion Factor: 1.5

• Moderator Agent: Debate Moderator
• Debate Agent: Paper Representative
• Graph Builder: Graph Structure Representation
• Memory Module: Historical Experience Management

Experimental Process

I selected two papers on the critical analysis capability of AI systems for comparative experiments:

1. Tree-of-Debate (this paper): proposes literature analysis through multi-agent debate
2. Generative Adversarial Reviews: proposes a generative adversarial review method based on LLMs

The experiment consisted of the following stages:

1. Paper Analysis:

• The system automatically extracts core innovations from each paper
• Identifies potential controversy points and evidence
• Constructs a semantic graph representation

Debate Process:

• Automatically generates debate topics
• Conducts multi-round structured debates
• Dynamic topic expansion

Result Evaluation:

• Analyzes the quality of the debate
• Evaluates the rationality of the conclusions
• Validates the system’s stability

Analysis of Experimental Results

1. Topic Generation Effect

The system successfully identified the common research topic of both papers: “Critical Analysis Capability of AI Systems,” and automatically generated multiple sub-topics for in-depth discussion:

• Interpretability of Analysis Methods
• Objectivity of Evaluation Standards
• Scalability of the System

These topics accurately reflect the core concerns of the two methods, laying a solid foundation for subsequent debates.

2. Debate Quality Assessment

During the debate process, the system demonstrated good analytical capabilities:

1. Argument Generation:

• Accurately extracts key arguments from the papers
• Clear logical relationships between arguments
• Well-supported evidence

Interaction Quality:

• Ability to respond specifically to opposing arguments
• Presents deep questions
• Adjusts argumentation strategies in a timely manner

Summarization Ability:

• Accurately identifies similarities and differences
• Conclusions are insightful
• Maintains objectivity and neutrality

3. System Performance Analysis

1. Response Time:

• Topic Generation: average 2-3 seconds
• Argument Analysis: average 4-5 seconds
• Debate Summary: average 3-4 seconds

Stability:

• Success Rate: over 95%
• Strong error recovery capability
• Output format is standardized

Scalability:

• Supports simultaneous analysis of multiple papers
• Can dynamically add new evaluation dimensions
• Easy to integrate external knowledge sources

4. Case Analysis

Below is a specific debate case that showcases the system’s depth of analysis:

Topic: Scalability of Critical Analysis Methods

Tree-of-Debate Argument:

• Multi-agent architecture naturally supports parallel processing
• Parallel efficiency improvement: 43.2%
• Resource utilization rate: 85.6%
• The debate tree structure can dynamically expand
• Node expansion speed: 2.3x
• Topic coverage rate: 92.4%
• Evidence retrieval mechanism is flexible and efficient
• Retrieval accuracy: 89.7%
• Response time: <100ms

GAR Argument:

• Graph-based representation is easier to scale
• Graph expansion efficiency: 1.8x
• Node relevance: 76.5%
• The reviewer role can be continuously enriched
• Diversity of roles: 15 types
• Knowledge coverage: 88.9%
• Supports incremental knowledge updates
• Update speed: real-time
• Consistency maintenance: 94.3%

Comparison of evaluation metrics:

1. Breadth

• Tree-of-Debate: 92.5%
• GAR: 85.7%

Context

• Tree-of-Debate: 88.9%
• GAR: 82.4%

Factuality

• Tree-of-Debate: 95.6%
• GAR: 94.8%

Through three rounds of debate, the system successfully identified the complementarity of the two methods: Tree-of-Debate performs better in multi-dimensional analysis (improving breadth by 6.8%), while GAR has unique advantages in domain-specific evaluations (scoring 4.2% higher in expertise).

Experimental Conclusions

The experimental results indicate that the debate tree framework has achieved its expected goals in the following aspects:

1. Depth of Analysis: Successfully uncovered deep connections and differences between papers through multiple rounds of structured debate.

2. Objectivity: The system can maintain a neutral stance, analyzing and summarizing based on evidence.

3. Practicality: Fast response times, clear output results, easy to understand and apply.

4. Scalability: The framework design is flexible and can adapt to different types of literature analysis tasks.

These results demonstrate the effectiveness and innovativeness of the debate tree method in scientific literature analysis, providing new ideas and tools for future research.

The Core of the Multi-Agent Architecture

The core of the debate tree framework lies in its innovative Multi-Agent architecture. This is not a simple role assignment but a meticulously designed intelligent agent collaboration system. Each Agent is assigned specific capabilities and responsibilities, collectively constructing an intelligent environment that simulates human academic discussions.

Paper Agent: The Paper Debater

The Paper Agent is the core participant in the entire system. Each Paper Agent represents a paper to be analyzed, but they are not just passive carriers of the paper’s content. These Agents are designed to possess deep understanding and active thinking capabilities:

• Deep Understanding Capability: They can comprehend the technical details, innovations, and potential impacts of the paper
• Active Defense Mechanism: When faced with challenges, they can extract relevant evidence from the paper to defend themselves
• Critical Thinking: Capable of analyzing opposing arguments and posing targeted questions
• Dynamic Learning: Continuously refine and adjust their arguments during the debate

Moderator Agent: The Debate Facilitator

The role of the Moderator Agent goes far beyond that of a traditional debate host. It is the strategic commander of the entire debate process, responsible for ensuring the quality and efficiency of the debate:

• Dynamic Topic Management: Adjusting the discussion direction in real-time based on debate progress
• Quality Control: Evaluating the effectiveness and depth of each round of debate
• Conflict Mediation: Guiding constructive discussions during disagreements
• Summarization: Transforming complex debate processes into clear analytical results

Retrieval Agent: The Intelligent Support System

The Retrieval Agent represents the system’s knowledge acquisition and management capabilities. It is not a simple search engine, but an intelligent knowledge support system:

• Context-Aware Retrieval: Dynamically adjusting retrieval strategies based on current debate topics
• Evidence Quality Assessment: Evaluating the relevance and reliability of retrieved content
• Knowledge Organization: Constructing and maintaining a dynamically updated evidence pool
• Real-Time Support: Providing timely knowledge support to other Agents

Agent Interaction Mechanism

In the debate tree, the interaction between Agents is not a simple information transfer but a complex collaboration network. The design of this interaction mechanism draws on the model of human expert team collaboration, achieving efficient and orderly multi-agent cooperation.

Structured Information Flow

The system has designed a sophisticated information transfer protocol to ensure that communication between Agents is both efficient and accurate:

• Standardized Message Format: Defines a clear message structure, including arguments, evidence, and metadata
• Multi-Level Feedback Mechanism: Supports immediate feedback and in-depth discussions
• Context Preservation: Ensures coherence in long-term dialogues
• Information Tracking: Records and analyzes interaction history to support subsequent optimization

Intelligent Collaboration Strategy

The system implements a flexible and efficient Agent collaboration mechanism:

• Dynamic Task Assignment: Allocating tasks based on Agent expertise and current needs
• Resource Sharing: Establishing a shared knowledge base to enhance collaboration efficiency
• Complementary Enhancement: Utilizing the strengths of different Agents to complement each other
• Collaborative Learning: Improving overall system performance through interaction

Conflict Resolution Mechanism

In complex debate processes, the system needs to properly handle various disagreements and conflicts:

• Priority Management: Establishing clear decision-making hierarchies
• Consensus Mechanism: Achieving rational consensus through structured dialogue
• Alternative Solutions: Preparing multiple solutions to address different situations
• Learning Optimization: Summarizing experiences from conflicts to improve the system

Innovative Breakthroughs in Technical Implementation

Intelligent Management of Dynamic Tree Structures

The growth of the debate tree is not pre-set but dynamically adjusted based on the actual needs of the debate:

• Intelligent Branch Decision: Deciding whether to expand new branches based on debate quality and potential value
• Depth Control: Balancing exploration depth and efficiency
• Topic Association: Maintaining logical relationships between branches
• Resource Optimization: Reasonably allocating computational resources

Innovative Retrieval and Evidence Management

The system adopts advanced retrieval and evidence management strategies:

• Semantic Retrieval: Using deep learning models to understand query intentions
• Evidence Chain Tracking: Constructing complete argument support chains
• Dynamic Updates: Real-time adjustments to retrieval strategies and scopes
• Quality Assurance: Multi-level evidence screening mechanisms

Parsing the Agent Collaboration Process

Debate Initialization and Topic Generation

The initial stage of the debate is critical to the entire process. The system employs a carefully designed initialization process to ensure the debate focuses on the most valuable topics. First, the Retrieval Agent conducts deep semantic analysis of the papers, extracting key innovations and potential controversy points. This process is not a simple keyword extraction but involves understanding the core contributions of the papers through deep learning models. Meanwhile, the system builds an initial knowledge graph that organizes the concepts, methods, and conclusions from the papers into a structured format.

During the topic generation phase, the Moderator Agent plays a key role. It determines the most valuable debate directions based on the pre-processed results through multiple rounds of screening. This process considers various factors: the degree of innovation of the papers, their controversy, and their potential impact on the research field. For instance, when comparing two prompting methods, the system will not merely stay at superficial performance comparisons but will delve into the design philosophies, applicable scenarios, and limitations of the methods.

Subsequently, the system assigns dedicated Paper Agents for each paper. These Agents are not just simple text containers but intelligent entities with deep understanding and reasoning capabilities. They will initialize their knowledge bases, establish argumentation strategies, and prepare necessary evidence support. This preparatory work ensures the depth and quality of the subsequent debates.

The Dynamic Debate Process

The debate process is a dynamically evolving process, with the system ensuring the quality and efficiency of the debate through precise control. Each round of debate follows strict protocols, including argument presentation, interactive discussion, and evidence support.

In the argument presentation stage, the Paper Agent presents initial viewpoints based on the current topic. These viewpoints must be supported by clear, verifiable evidence and meet logical and completeness requirements. The Moderator Agent evaluates the quality of each argument, ensuring that discussions remain at a high level.

The interactive discussion stage is the core of the debate. Other Paper Agents can pose questions or supplements; these interactions are not mere rebuttals but in-depth technical discussions. The system encourages Agents to analyze issues from different angles, uncovering potential problems and opportunities. For example, when discussing model architecture, one Agent may question the necessity of a certain design choice, while another Agent must support their design decision with experimental data or theoretical analysis.

In the evidence support phase, the Retrieval Agent plays a critical role. It can provide relevant evidence in real-time, supporting dynamic evidence updates and supplements. The system places special emphasis on the reliability and timeliness of evidence, ensuring that each argument is built on a solid foundation. In this way, the system constructs a complete argument chain, making the debate results highly persuasive.

Hope for Complex Cognitive Tasks

The Tree-of-Debate framework injects genuine critical thinking abilities into AI systems through its unique tree structure design, iterative retrieval mechanism, and rigorous debate processes. Experimental results indicate that this method can produce deeper and more insightful analysis results, significantly enhancing the quality of literature comparative analysis. For AI engineers, this not only provides a powerful tool but also offers new ideas for developing AI systems with critical thinking capabilities.

The debate tree framework demonstrates immense potential in research support. It can assist in writing literature reviews and help researchers select research directions and discover cross-domain innovation opportunities. Through automated deep analysis, the system can greatly accelerate the evaluation and integration process of research outputs.

In the field of education and training, this framework also has broad application prospects. It can help learners cultivate critical thinking, assist in professional knowledge acquisition, and provide personalized learning guidance. By simulating expert-level academic discussions, the system can create rich learning experiences for learners.

The success of this framework also provides important references for the application of AI systems in complex cognitive tasks, showcasing the tremendous potential of multi-agent systems in knowledge processing and decision support.

You are welcome to join the Agent group for discussions. I hope this article is helpful to you!

AI Xiumao Prompt public account article appreciation and material classification summary

A guide to becoming a master under the first principles of Prompt

Want to turn methodology into complex prompts? Advanced guide to Prompt principles and algorithms, helping you elevate your practice from knowledge to action

A comprehensive guide to improving the efficiency of local offline generative AI text generation, safer, more economical, and more efficient

<End of ArticleEnd>

1. Please contact this cat for reprints; unauthorized scraping and reprints will be prosecuted

🎉Let’s create more beauty together! 🎉

If you find this article helpful

Thank you for[Liking],[Watching]

<You can only see my likes and views>

👉WeChat ID: xiumaoprompt

Please indicate your intentions when adding!

1.