Why Do Multi-Agent LLM Systems Fail? #

1. TL;DR #

MAS fail at 41–86.7% on SOTA benchmarks. Grounded-Theory analysis of 150+ traces yields MAST: 14 failure modes in 3 categories (system design, inter-agent misalignment, verification). An o1 LLM-as-Judge ($\kappa=0.77$) scales this to 1642 traces. Failures are structural — fixable with the same model, but isolated fixes only add +9.4–15.6%.

2. Q1 / Q2 / Q3 #

Q1 — 痛点 (pain point) #

Multi-agent LLM systems (MAS) are widely adopted for task decomposition, parallelization, context isolation, and multi-perspective reasoning, yet their measured gains over single-agent frameworks or simple best-of-N baselines are often minimal. Empirically, seven SOTA open-source MAS fail on 41% to 86.7% of tasks (Figure 5). The field lacks any principled answer to why: failures in MAS are hard to root-cause (convoluted agent interactions, compounding of individual model errors with system-design flaws), and there is no standardized vocabulary or dataset to make failure analysis consistent across systems.

Q2 — 方法 (method) #

Three coupled artifacts, built bottom-up rather than top-down:

1. MAST-Data — 1642 annotated execution traces from 7 MAS frameworks (ChatDev, MetaGPT, HyperAgent, AppWorld, AG2, Magentic-One, OpenManus), spanning coding / math / general-agent tasks and four model families (GPT-4 series, Claude 3.x, Qwen2.5-Coder, CodeLlama).
2. MAST — the taxonomy itself: 14 fine-grained failure modes clustered into 3 categories, mapped to execution stages (Pre / Execution / Post). Derived via Grounded Theory (open coding → constant comparison → memoing → theorizing to theoretical saturation) over 150+ traces by 6 human experts, then hardened via 3 rounds of inter-annotator agreement to $\kappa=0.88$.
3. LLM-as-a-Judge annotator — an o1-based pipeline (MAST definitions + few-shot human examples) that reproduces human labels at 94% accuracy / $\kappa=0.77$, validated out-of-domain at $\kappa=0.79$, packaged as pip install agentdash.

核心技术壁垒: the hardest-to-replicate asset is not any algorithm but the empirically saturated, IAA-validated taxonomy. Reaching $\kappa=0.88$ across six experts on 15,000-line traces via Grounded Theory (>20 hrs/expert on the initial 150) is what makes the 14-mode schema stable enough that an LLM judge can then be calibrated to it — the taxonomy is the load-bearing insight, and it took hundreds of hours of expert annotation to converge.

Q3 — 结果 (results) #

• 14 modes with prevalence over 1642 traces; the three most common are FM-1.3 Step repetition (15.7%), FM-2.6 Reasoning-action mismatch (13.2%), FM-1.5 Unaware of termination (12.4%).
• Category correlations are low (0.17–0.32), confirming the three categories capture distinct root causes.
• MAST-guided interventions on the same model yield +9.4% (ChatDev role-spec fix) and +15.6% (ChatDev task-objective verification), but task completion stays low — isolated fixes are insufficient.
• Model & architecture matter but non-monotonically: MetaGPT has 60–68% fewer FC1/FC2 failures than ChatDev yet 1.56× more FC3 (verification) failures.

3. 架构 / 方法图 #

The taxonomy structure #



Paper's Figure 1, verbatim (caption: "MAST: A Taxonomy of MAS Failure Modes. The inter-agent conversation stages indicate when a failure typically occurs within the end-to-end MAS execution pipeline... The percentages shown represent the prevalence of each failure mode and category as observed in our analysis of 1642 MAS execution traces.").

This is the central artifact: the 14 modes are laid out against three execution stages (Pre-Execution, Execution, Post-Execution) and three categories (System Design, Inter-Agent Misalignment, Task Verification), with each mode annotated by its prevalence. A mode spanning multiple stages signals its root cause can surface at different points in the pipeline — the reader should notice that verification failures (FC3) concentrate post-execution while design failures (FC1) originate pre-execution.

The construction workflow #



Paper's Figure 2, verbatim (caption: "Methodological workflow for constructing the MAST-Data dataset, involving the empirical identification of failure modes, the development of MAST, iterative refinement through inter-annotator agreement studies ($\kappa=0.88$), and the creation of a scalable LLM annotation pipeline.").

The pipeline reads: collect 150 traces → GT open coding by 6 experts → draft MAST → 3 IAA rounds to $\kappa=0.88$ → calibrate o1 LLM-judge to $\kappa=0.77$ → validate out-of-domain ($\kappa=0.79$) → scale to the full 1642-trace MAST-Data. The key structural point is that the human effort is front-loaded into taxonomy construction, after which the LLM judge amortizes annotation cost.

The agent-loop lens (failure-mode placement) #

The taxonomy is best read against one MAS turn: where in observation → plan → act → verify → terminate does each mode strike?

stateDiagram-v2 [*] --> Spec: task + role prompts Spec --> Plan: FM-1.1 disobey task / FM-1.2 disobey role Plan --> Act: FM-2.3 derailment / FM-2.6 reason-action mismatch Act --> Communicate: FM-2.2 no clarification / FM-2.4 withholding / FM-2.5 ignore input Communicate --> Verify: FM-1.3 step repetition / FM-1.4 history loss / FM-2.1 reset Verify --> Terminate: FM-3.2 no verification / FM-3.3 incorrect verification Terminate --> [*]: FM-1.5 unaware of termination / FM-3.1 premature termination

This redraw is justified because the paper has no single figure showing the per-turn placement of every mode; it maps each of the 14 modes onto the canonical agent loop, making explicit that misalignment (FC2) modes cluster around the act/communicate transition where inter-agent information must flow.

Interaction pattern & scope #

Scope is diagnostic, not a new agent: it studies open-ended MAS (coding, math, general web/file tasks) under multi-turn, fully autonomous orchestration. The architectures analyzed span assembly-line (MetaGPT), hierarchical workflow (ChatDev, HyperAgent, OpenManus), and star topology (AppWorld, Magentic-One) — the taxonomy is deliberately architecture-agnostic.

4. 作者证明 #

无形式化作者证明 — 仅实证. The paper offers no convergence, regret, or success-rate bound. Its "model" is the empirical taxonomy, and validity is argued statistically rather than proved. What could have been bounded: a formal guarantee that the 14 modes are collectively exhaustive (they explicitly disclaim this: "we do not claim it covers every potential failure pattern"), or a PAC-style bound on LLM-judge agreement with the true human label distribution.

Notation used in the empirical claims:

Symbol	Meaning
$\kappa$	Cohen's Kappa inter-annotator / human-vs-judge agreement
FC1/FC2/FC3	the three failure categories
FM-x.y	failure mode y within category x
1642	total annotated traces in MAST-Data

6 empirical checks (in lieu of a formal proof):

1. Taxonomy reliability: human IAA converges to $\kappa=0.88$ over 3 rounds — verifiable against the reported 15-trace-per-round protocol.
2. Judge calibration: o1 few-shot reaches accuracy 0.94, $\kappa=0.77$ (Table 2); zero-shot only $\kappa=0.58$ — few-shot is load-bearing.
3. Out-of-domain generalization: applying finalized MAST to unseen MAS (OpenManus, Magentic-One) + benchmarks (MMLU, GAIA) still yields $\kappa=0.79$.
4. Category distinctiveness: pairwise category correlations 0.17–0.32 (Fig 6); fine-grained mode correlations peak at 0.63 (Fig 7) — categories are near-orthogonal, modes only mildly conflatable.
5. Success-rate sweep (empirical, per deep-agent asks): the axes are (MAS architecture × backbone model × benchmark difficulty). Monotonicity holds along difficulty (GSM < MMLU < Olympiad failure rates in Table 8) and along model quality (GPT-4o < Claude on FC1 in MetaGPT; Qwen ≪ CodeLlama in Table 6). It does not hold cleanly along architecture: MetaGPT wins FC1/FC2 but loses FC3.
6. Intervention effect: MAST-guided fixes measurably reduce targeted modes (Fig 10/11) and lift accuracy (+9.4%, +15.6%), with Wilcoxon significance under GPT-4o (p=0.03) — but not always under GPT-4 (topology p=0.4).

Failure-mode classification (deep-agent ask): 14 classes; the dominant single mode is FM-1.3 Step repetition (15.7%). The paper is diagnostic — it does not target one class, it argues the whole distribution requires structural (not tactical) remedies.

5. 实验与数据 #

Configuration matrix #



Paper's Table 1, verbatim (dataset configuration details; HE=Human Evaluated, HA=Human Annotated, LA=LLM Annotated).

Table 1 is the backbone of every downstream claim: it shows the 18 configuration rows summing to 1642 traces, and reveals the deliberate crossing of model family (GPT-4o / Claude-3.7 / Qwen2.5-Coder / CodeLlama) against framework (ChatDev / MetaGPT) on ProgramDev-v2 — this is what enables the model-vs-architecture disentanglement in §5.

Failure rates that motivate the paper #



Paper's Figure 5, verbatim (caption: "Failure rates of six popular Multi-Agent LLM Systems with GPT-4o and Claude-3.7-Sonnet. Performances are measured on different benchmarks, therefore they are not directly comparable.").

This is the source of the headline "41–86.7%" claim. The reader should note the caveat: benchmarks differ per system, so this figure motivates the existence of a problem, not a cross-system ranking.

Per-system failure profiles #



Paper's Figure 4, verbatim (caption: "Distribution of failure in MAST-Data with MAST labels on total 210 traces... intended to illustrate system-specific failure profiles rather than to serve as a performance comparison across MAS.").

Figure 4 exposes system-specific fingerprints: AppWorld's star topology drives premature termination (FM-3.1); OpenManus over-repeats steps (FM-1.3); HyperAgent is dominated by step repetition + incorrect verification. These profiles are what make MAST actionable as a debugging tool.

Architecture vs model disentanglement #



Paper's Figure 9, verbatim (caption: "Comparison on MAST failure modes and categories on ProgramDev-v2... to analyze MAS architecture effect. GPT-4o is used on two comparative cases, one using ChatDev and the other on MetaGPT.").

Holding the model fixed at GPT-4o, MetaGPT has far fewer FC1/FC2 failures (its SOP encoding enforces spec/role adherence) but more FC3 failures, because ChatDev's explicit review+test phases buy verification quality. This is the paper's cleanest evidence that architecture choice trades one failure category for another.

Intervention accuracy #



Paper's Table 5, verbatim (Case Studies Accuracy Comparison, percentages).

Table 5 quantifies the ceiling of tactical fixes: on ChatDev ProgramDev-v0 the new topology lifts accuracy 25.0 → 40.6%, but on saturated HumanEval (89.6%) gains are marginal, and on AG2 GSM-Plus the improved prompt (89.75%) beats the new topology (85.50%) under GPT-4 — the very tension flagged as an open surprise (topology "more effective" in §H.3 by mode-count yet not always by accuracy).

6. 论证链 #

Step	Claim	Support (paper-internal)
1	MAS underperform and fail at 41–86.7%, so the "why" matters	Figure 5 failure rates; §1 gap vs single-agent/best-of-N
2	Failures are not random; recurrent patterns exist across systems	Grounded-Theory saturation over 150+ traces, 6 experts (§3.1)
3	Those patterns can be codified into a reliable taxonomy	3 IAA rounds reach $\kappa=0.88$ (§3.2); categories correlate only 0.17–0.32 (Fig 6)
4	The taxonomy scales via an LLM judge without losing fidelity	o1 few-shot: 0.94 acc, $\kappa=0.77$ (Table 2); $\kappa=0.79$ out-of-domain (§3.4)
5	Applying it reveals failures stem from system design, not just model limits	Same-model interventions gain +9.4% and +15.6% (§4/§5.3)
6	But design failures need structural, not tactical, remedies	interventions leave completion low; topology > prompt yet both insufficient (§5.3, Appendix H)

7. 实现 cross-reference #

The taxonomy and LLM annotator are open-sourced. Concrete implementation surfaces:

• LLM-annotator library — pip install agentdash; entry point from agentdash import annotator, class annotator(openai_api_key), method .produce_taxonomy(trace) returning a dict with failure_modes, summary, task_completion, total_failures (Appendix C, verbatim usage snippet in L1).
• Repo / dataset — https://github.com/multi-agent-systems-failure-taxonomy/MAST; https://huggingface.co/datasets/mcemri/MAST-Data. ProgramDev at traces/programdev/programdev_dataset.json.
• ChatDev intervention — role prompts before/after in Appendix M (M.1 initial, M.2 modified); the topology fix converts ChatDev's DAG to a cyclic graph terminating only on CTO confirmation with an iteration cutoff (Appendix H.2).
• AG2 intervention — three-role topology (Problem Solver / Coder / Verifier, only Verifier can terminate) with prompts in Appendix L.3–L.5.

核心技术壁垒 (§7 elaboration): replicating this work is not a coding exercise — the agentdash judge is trivial to pip install, but reproducing the taxonomy behind it requires re-running Grounded Theory to theoretical saturation with multiple expert annotators on multi-thousand-line traces and hitting $\kappa=0.88$. The barrier is annotation methodology and human agreement, not software. Without a saturated, high-$\kappa$ taxonomy, any LLM judge calibrated to it inherits noise.

关键实现细节 (easy-to-miss tricks):

1. Few-shot examples are not optional garnish — they lift the o1 judge from $\kappa=0.58$ to $\kappa=0.77$ (Table 2); the few-shot set is drawn from the human-annotated Appendix N traces, one per mode.
2. Annotation cost scales with trace length, not task count: OpenManus costs $4.14/trace vs AppWorld's $0.37 (Table 9) purely because hierarchical frameworks emit verbose logs — a hidden budget line when scaling annotation to new frameworks.