This article discusses Anthropic’s agent evaluation, production reliability, and engineering challenges.

Agent Evaluation

How do we conduct agent evaluation? The traditional approach is “given input X, one must follow steps Y to achieve the correct output Z.” However, multi-agent systems cannot be evaluated this way because they do not have a fixed, unique, pre-defined problem-solving path.

What to Evaluate

Instead of checking whether the pre-defined path was followed, we should assess whether there is a reasonable process and correct results.

• • Result Orientation: Is the final answer correct and does it meet user needs?
• • Process Rationality: While variations in paths are allowed, it is essential to ensure that the paths are logical, efficient, cost-effective, and reliable.

Start Evaluation with Small Sample Sets

Do not wait until the data volume is large to start evaluations; begin evaluating with small samples.

LLM as Judge

• • Research Output Evaluation Challenges: Research outputs are often free-form text, making them difficult to evaluate programmatically, as they rarely have a single correct answer.
• • Suitability of LLM as an Evaluation Tool: Due to its powerful language understanding and generation capabilities, LLMs are well-suited for evaluating such complex research outputs.

Scoring Criteria

Anthropic has established several scoring criteria:

• • Factual Accuracy (Does the statement align with the source?)
• • Citation Accuracy (Does the cited source genuinely support the statement?)
• • Completeness (Does it cover all required content?)
• • Source Quality (Does it prioritize high-quality primary sources over low-quality secondary sources?)
• • Tool Efficiency (Were appropriate tools used, and were the number of calls reasonable?)

Scoring Method: The LLM evaluation tool scores each output based on the above criteria, with a scoring range from 0.0 to 1.0, and provides a pass or fail grade.

Experiments and Optimization

• • Multi-Judge Experiments: Initially, multiple LLM evaluation tools were used to assess each criterion, but this method proved inconsistent and did not align well with human judgment results.
• • Single-Judge Optimization: Ultimately, it was found that using a single LLM evaluation tool with a single prompt to output scores and pass/fail grades was the most consistent and closely aligned with human judgment results.

The Importance of Human Evaluation

1. 1. Limitations of Automated Evaluation: Automated evaluation tools (such as LLM evaluation tools) are efficient but may overlook edge cases. These include:

• • Hallucinated Answers: In unusual queries, agents may generate seemingly reasonable but actually incorrect answers (hallucinated answers).
• • System Failures: Automated evaluation may fail to detect potential faults in the system.
• • Subtle Source Selection Bias: Automated evaluation may not recognize subtle biases in the agent’s choice of information sources.

2. Advantages of Human Evaluation: Human evaluation can compensate for the shortcomings of large model evaluations, identifying extreme cases that evaluation scripts may miss, such as hallucinated answers under rare queries, system failures, or subtle biases in source selection. For example, Anthropic found that agents consistently prioritized SEO-optimized content while skipping authoritative but lower-ranked academic PDFs or personal blogs. They later incorporated heuristics for source quality into their prompts to resolve this issue.

3. Complexity of Multi-Agent Systems

• • Multi-agent systems exhibit emergent behaviors, which are not produced through specific programming but are the result of interactions between agents. For instance, a small change to the lead agent may unpredictably alter the behavior of subagents.
• • The key to success lies in understanding the interaction patterns between agents, rather than just the behavior of individual agents.

Production Reliability and Engineering Challenges

In traditional software, a small bug may only cause a function to crash, degrade performance, or trigger a single failure.

In agentic systems, small changes can snowball, leading to significant behavioral drift—making it exceptionally challenging to write reliable code for complex agents that need to maintain state over long periods.

1. Adding Fault Recovery Functionality

Agents may run for extended periods, maintaining state between multiple tool calls. This means ensuring the code executes persistently and handles errors appropriately at every step.

When errors occur, one cannot simply “restart”: restarting is too costly. Therefore, Anthropic built a system capable of recovering from fault points in place, incorporating retry logic and regular snapshots to ensure breakpoint recovery functionality.

2. Developing Full Link Tracing

In traditional software, the same input generally yields the same output, but AI agents are “dynamic and non-deterministic”—running the same set of prompts twice may result in different internal decision paths, significantly increasing debugging difficulty. For example, a user might complain, “The agent can’t find obvious information.” Just relying on logs won’t reveal where the problem lies—did it generate poor search keywords? Did it select a junk webpage? Or did the tool itself fail to call?

Anthropic introduced a “full production tracing” system that records every call, every decision node (keywords → results → scores → next actions), without recording the dialogue text, ensuring user privacy.

Benefits:

• • It allows one to quickly identify the root cause, such as “Oh, the failure was because it parsed the PDF as a webpage, leading to failure”;
• • It can uncover unexpected anomalies, such as “two subagents repeatedly passing the buck”;
• • By categorizing common failures into patterns, one can systematically fix bugs, adjust prompts, and supplement tools, rather than relying on guesswork.

3. Considering User Activity During Deployment

Given that updates can be released at any time while users are using the agent, Anthropic employs “rainbow deployments”: both the old and new versions are online simultaneously, gradually shifting traffic from the old instance to the new instance to avoid interrupting running agents.

4. Synchronous vs. Asynchronous Execution

Advantages of Synchronous Execution

• • Simple Process: The lead agent waits for all subagents to return in a fixed order, resulting in lower scheduling and debugging costs.
• • Consistent State: All sub-tasks are completed simultaneously, and outputs are naturally aligned, eliminating the need for additional merging steps.

Challenges of Synchronous Execution

• • Throughput Limitations: Any delay from a subagent slows down the entire system, limiting overall parallelism.
• • Lack of Flexibility: The lead agent cannot dynamically adjust the goals of subagents during execution; subagents also cannot share information or collaborate in real-time.
• • Resource Idling: When most subagents have completed while a few are still executing, computational power and bandwidth are blocked, leading to decreased utilization.

Advantages of Asynchronous Execution

• • Significantly improved parallelism, as each subagent can independently progress and derive new tasks as needed;
• • The lead agent can receive intermediate results in real-time, allowing for timely redirection or termination of sub-tasks, achieving fine-grained control;
• • Subagents can collaborate instantly through messaging mechanisms, enhancing overall efficiency.

Challenges of Asynchronous Execution

• • Result Aggregation: Additional mechanisms are needed to sort, deduplicate, and aggregate results that arrive out of order;
• • State Consistency: Distributed snapshots or transaction protocols are required to ensure global view synchronization, increasing system complexity;

5. Context Management

When an AI agent needs to engage in hundreds of rounds of dialogue with users, how can it “remember” what has happened before without failing due to context window overflow?

1. 1. Solution Approach:

• • Compression: Summarize completed dialogues into a few sentences.
• • External Memory: Store summaries, key data, research plans, etc., in external memory (databases, files, vector indexes), rather than cramming everything into the dialogue.

2. Specific Implementation: After completing a “work phase” (e.g., gathering reasons for chip shortages or writing the first chapter of a business plan), the main agent first summarizes itself, then writes the summary and core results into external memory. When the dialogue is about to fill the window:

At any subsequent moment, the agent can retrieve the research plan, data tables, citation lists from external memory as needed, without having to keep all this bulky information in the dialogue context.

• • Initiate a new sub-agent (clean context, ample tokens).
• • Hand-off: Transfer the summary from external memory + next task to the new sub-agent, allowing for seamless continuity, so the user hardly notices the “change of personnel.”

Conclusion

Transforming an AI prototype into a truly reliable production system is much more challenging than most people imagine.

1. 1. In AI Agent projects, running a demo only accounts for 40% of the workload; the remaining 80% must be spent on the “last mile”—ensuring it runs reliably and scalably in a production environment over the long term.
2. 2. Running on a developer’s computer does not equal running in a production environment. To turn a demo into a 24/7 available service requires additional engineering: persistence, retries, concurrency, monitoring, gray releases, rollbacks, cost control… all of which are already complex in traditional software, and even more so in “thinking” AI agents.
3. 3. In traditional software, a small bug typically only affects one function; in AI agents, a small error can be magnified by the “inference chain,” leading to deviations in all subsequent steps. For example, if a search returns incorrect information, the agent may make entirely erroneous business decisions based on that.
4. 4. Many teams underestimate this gap, leading to project delays or failures. Therefore, engineering investments should be planned in advance, rather than “let’s just get it done first.”