Reflexion: Language Agents with Verbal Reinforcement Learning
Reflexion: Language Agents with Verbal Reinforcement Learning — L2 #
§1 TL;DR #
Reflexion turns a sparse reward (binary/scalar) into a verbal self-reflection stored in an episodic memory buffer, so an LLM agent improves across trials without any weight update. Policy is $\theta = \{M_a, mem\}$. Hits 91% HumanEval pass@1, +22% AlfWorld, +20% HotPotQA — but only on strong backbones.
Agent scope: closed-to-open task class spanning three families — sequential decision-making (AlfWorld, open-ish), multi-hop reasoning/RAG (HotPotQA), and single-function code generation (HumanEval/MBPP/Leetcode). Interaction pattern is multi-turn within a trial, iterated across trials (trial-and-error episodes). Autonomy is fully autonomous: the agent supplies its own evaluation signal (LLM classifier, heuristic, or self-generated unit tests) — no human-in-the-loop.
§2 Q1 / Q2 / Q3 #
Q1 — 痛点 (pain point). LLM agents built on ReAct / SayCan / Toolformer can act in environments but cannot cheaply learn from failure. Traditional RL fixes this via gradient descent on weights, which for a large LLM demands "substantial amounts of compute and time" plus fine-tuning. So in-context examples were the only teaching channel — and a scalar reward gives no actionable credit assignment ("which action was the mistake?").
Q2 — 方法 (method). Convert the reward into natural-language feedback that acts as a semantic gradient. Three modular models cooperate: an Actor $M_a$ (an LLM prompted as CoT or ReAct) samples action $a_t \sim \pi_\theta$; an Evaluator $M_e$ scores the trajectory into a scalar $r_t$; a Self-Reflection model $M_{sr}$ reads $\{\tau_t, r_t\}$ and emits a verbal lesson $sr_t$. The lesson is appended to a bounded memory buffer $mem$ (capacity $\Omega \in [1,3]$), which becomes the long-term component of the policy. Next trial, the Actor conditions on $mem$ and improves.
The 核心技术壁垒 (core moat): the verbal credit-assignment step itself — $M_{sr}$ inferring a counterfactual ("action $a_i$ should have been $a_i'$, which would have led to $a_{i+1}'$") and phrasing it as a first-person self-hint that measurably re-steers the next trajectory. This is what separates Reflexion from blind test-driven retry (which the ablation shows gives zero gain). It only emerges on strong backbones — the hardest thing to replicate is not the loop plumbing but the backbone's latent self-diagnosis ability.
Q3 — 结果 (results). SOTA on almost all code benchmarks: HumanEval Python 91% vs GPT-4's 80.1%; Leetcode Hard Python 15% vs 7.5%; Rust HumanEval 68% vs 60%. AlfWorld 130/134 tasks (+22% absolute over ReAct-only, which stalls at trials 6–7 and a 22% hallucination floor). HotPotQA +20%; self-reflection alone adds +8% over episodic-memory-only. Candidly reports failures: underperforms on MBPP Python (0.77 < 0.80 baseline) and no gain on WebShop or on a weak backbone (starchat-beta 0.26 → 0.26).
§3 架构 / 方法图 #
The system diagram and its companion algorithm are the paper's own Figure 2.
Paper's Figure 2, verbatim (caption: "(a) Diagram of Reflexion. (b) Reflexion reinforcement algorithm").
Panel (a) shows the closed loop: the Actor produces a trajectory into the environment, the Evaluator maps the trajectory to a reward, and the Self-Reflection model amplifies that reward into text stored in memory; the Actor re-reads memory on the next trial. Panel (b) is Algorithm 1. Notice that the two memory tiers are structurally distinct — short-term memory is the current trajectory $\tau_t$ living in the context window, while long-term memory $mem$ is the accumulated list of verbal reflections.
The task coverage that this single loop generalizes over is the paper's Figure 1.
Paper's Figure 1, verbatim (caption: "Reflexion works on decision-making 4.1, programming 4.3, and reasoning 4.2 tasks").
The key point of Figure 1 is that one mechanism (trial → reflect → retry) is instantiated three ways only by swapping the Evaluator (heuristic/LLM classifier for AlfWorld, exact-match for HotPotQA, self-generated unit tests for code) — the Actor and Self-Reflection machinery is shared.
The agent loop (one turn / one trial):
- Memory model. Short-term = context-window trajectory; long-term = sliding window of the last $\Omega$ reflections (1 for code, 3 for AlfWorld/HotPotQA); episodic = the trajectory log fed to $M_{sr}$.
- Tool invocation. Delegated to the Actor's underlying policy (ReAct emits
search[...]/Finish[...]; code agent emits a function body). Reflexion adds no new tool syntax. - Error recovery. On failure the agent does not backtrack within a trajectory; it resets the environment and starts a fresh trial with the failure distilled into
mem. The fallback state is "empty environment + accumulated lessons." - Planning style. ReAct-style think-then-act within a trial; across trials it is an outer optimization loop, not tree search — no branch revisiting, no undo of a committed action.
§4 作者证明 #
无形式化作者证明 — 仅实证. The paper explicitly states it has "not having a formal guarantee for success" and frames Reflexion as policy optimization that "may still succumb to non-optimal local minima." What could have been bounded: a convergence rate of success-rate vs trial count, or a regret bound on the outer loop — neither is attempted.
Notation table (the loop is nonetheless formalized in Algorithm 1):
| Symbol | Meaning |
|---|---|
| $M_a, M_e, M_{sr}$ | Actor / Evaluator / Self-Reflection models |
| $a_t \sim \pi_\theta$ | action/generation sampled from policy at time $t$ |
| $o_t$ | environment observation at time $t$ |
| $\tau_t$ | trajectory $[a_0, o_0, \dots, a_i, o_i]$ of trial $t$ |
| $r_t$ | scalar reward for trial $t$ |
| $sr_t$ | verbal self-reflection summary for trial $t$ |
| $mem$ | long-term memory buffer of reflections |
| $\Omega$ | max stored experiences (1–3) |
| $\theta = \{M_a, mem\}$ | policy parameters = Actor LLM + memory |
Physical meaning of the two load-bearing equations: the reward is produced by applying the Evaluator to a trajectory, $r_t = M_e(\tau_0)$ — reward is computed by a model on the whole trajectory, not a hand-designed dense signal. And $\pi_\theta(a_i \mid s_i)$ with $\theta = \{M_a, mem\}$ reframes prompt+memory as RL policy parameters, so "learning" = editing text in $mem$, not gradients.
Agent-specific checks (reproduced from the empirical sweep):
- Success-rate sweep — task family axis. Monotone improvement across trials on all three families (AlfWorld to near-100%, HotPotQA +20%, HumanEval 80→91). ✓
- Success-rate sweep — backbone axis. Table 6 sweeps text-davinci-003 / gpt-3.5-turbo / gpt-4 × {CoT(GT), ReAct}; Reflexion gain is positive in every cell (e.g. gpt-4 CoT(GT) 0.68 → 0.80). ✓ but non-uniform.
- Backbone strength threshold. starchat-beta (weak) gives 0.26 → 0.26 — the method fails below a capability threshold. Non-monotone claim substantiated: the self-correction ability is emergent. ✓
- Failure-mode classification. AlfWorld failures classed by reason (Figure 3b); dominant baseline failure = hallucinating possession of an item, which Reflexion's distilled self-hints "eliminate almost all of." The method is targeted at exactly this class. ✓
- Component ablation (is reflection load-bearing?). Table 4 Rust: removing self-reflection → 0.60 (no gain over base); removing test generation → 0.52 (below base). Reflection is the causal component. ✓
- Latency budget per turn. Not claimed as "interactive"; each trial = full trajectory + Evaluator pass + reflection generation. Cost scales with trials × (Actor tokens + reflection tokens); the paper caps memory at $\Omega$ and truncates to control prompt length. ✓ (implicit, no numeric latency).
- Memory is tiny and truncated. $\Omega = 1$ for code, $3$ for AlfWorld/HotPotQA — the buffer is deliberately kept to 1–3 reflections to fit context, and older reflections are dropped (sliding window). Success does not need a large memory.
- The code Evaluator gates on a syntactically-validated, AST-parsed, ≤6-test self-generated suite ($T = \{t_0,\dots,t_n\}$, $n \le 6$), and the design prefers false negatives over false positives — a false negative lets the agent keep correct code via reflection, while a false positive causes premature (wrong) submission. This asymmetry is the quiet reason HumanEval (1.4% FP) beats MBPP (16.3% FP).
§5 实验与数据 #
Decision-making (AlfWorld).
Paper's Figure 3, verbatim (caption: "(a) AlfWorld performance across 134 tasks ... using self-evaluation techniques of (Heuristic) and (GPT) ... (b) Classification of AlfWorld trajectories by reason of failure").
The (a) curve shows the diagnostic signature of learning: an immediate spike between the first two trials, then a steady climb over 11 more trials to near-perfect, whereas ReAct-only flatlines after trial 6–7. Panel (b) is why: Reflexion collapses the "hallucination / inefficient planning" failure bars that pin ReAct at a 22% hallucination floor.
Reasoning (HotPotQA).
Paper's Figure 4, verbatim (caption: "Reflexion improves search, information retrieval, and reasoning capabilities on 100 HotPotQA questions. (a) Reflexion ReAct vs Reflexion CoT (b) Reflexion CoT (GT) for reasoning only (c) Reflexion vs episodic memory ablation").
Panel (c) is the single most important experiment for the thesis: episodic memory (just re-feeding the last trajectory) is compared against full self-reflection, and reflection adds +8% absolute on top — evidence that the verbal explanation, not mere memory of the past attempt, does the work. Panel (b) shows even with ground-truth context the baseline misses 39% of questions, of which Reflexion recovers 14%.
Programming (the SOTA headline). Table 2 (pass@1) and Table 3 (test-generation confusion matrix) carry the code claims:
| Benchmark (Lang) | SOTA pass@1 | Reflexion pass@1 |
|---|---|---|
| HumanEval (PY) | 80.1 (GPT-4) | 91.0 |
| HumanEval (RS) | 60.0 | 68.0 |
| MBPP (PY) | 80.1 | 77.1 |
| MBPP (RS) | 70.9 | 75.4 |
| Leetcode Hard (PY) | 7.5 | 15.0 |
The MBPP-Python regression (0.77 < 0.80) is explained by Table 3: MBPP Python has a 16.3% false-positive unit-test rate (tests pass on a wrong solution → premature submit) vs only 1.4% for HumanEval Python. The whole code method is only as trustworthy as the self-generated test suite.
Ablation (Table 4, HumanEval Rust, 50 hardest, GPT-4). Base 0.60 → drop test-gen 0.52 → drop reflection 0.60 → full Reflexion 0.68. This is the sharpest evidence that blind trial-and-error debugging without verbal reflection is worthless on hard problems.
§6 论证链 #
| # | Step (paper-internal) | Support |
|---|---|---|
| 1 | Scalar/binary rewards give poor credit assignment; fine-tuning an LLM per task is too expensive. | §1 motivation; Sutton & Barto credit-assignment framing |
| 2 | Therefore amplify the reward into verbal feedback and store it as memory, treating prompt+memory as the policy $\theta=\{M_a,mem\}$. | §3, Algorithm 1 |
| 3 | Instantiate the same loop three ways by only swapping the Evaluator (heuristic/LLM, exact-match, self-generated tests). | §4.1–§4.3 |
| 4 | Across all three, Reflexion improves over strong baselines that plateau (AlfWorld +22%, HotPotQA +20%, HumanEval +11%). | Figs 3–4, Tables 2–3 |
| 5 | The verbal reflection — not memory-of-trajectory nor test-retry — is the causal ingredient. | Fig 4c (+8% over EPM); Table 4 (reflection-off = no gain) |
| 6 | The ability is emergent in strong backbones and bounded by self-evaluation reliability, so no formal guarantee is claimed. | Table 5 (starchat 0.26→0.26); MBPP false positives; §5 |
§7 实现 cross-reference #
Official implementation is open-sourced: [实现未公开] is not applicable — the paper releases code, demos, and datasets at https://github.com/noahshinn024/reflexion (§8 Reproducibility, Appendix C). File:line citations are unavailable from the L1 alone, so the anchors below are the verbatim prompt/algorithm artifacts the repo implements.
核心技术壁垒 (deep-dive). The single hardest-to-replicate insight is the verbal counterfactual credit-assignment performed by $M_{sr}$. Concretely, given a failed trajectory the model must (a) localize the mistaken action $a_i$ among many, (b) synthesize a corrected plan $a_i' \to a_{i+1}'$, and (c) phrase it as a first-person lesson that the Actor will actually follow next trial. Appendix B.1's "examine the mug with the desklamp" trajectory is the canonical example: the reflection "I should have looked for the desklamp first, then the mug" flips a fail into a success on trial 2. The barrier is that this requires the backbone to self-diagnose — an emergent capability absent in weak models (starchat-beta shows no gain). No amount of loop engineering recovers it below the capability threshold.
关键实现细节 (easy-to-miss tricks):
Negative-result artifact. The WebShop failure (Appendix B.2) is worth embedding: the agent shows no improvement after 4 trials and emits unhelpful reflections on tasks needing diverse exploration.
Paper's WebShop appendix figure, verbatim (caption: "Reflexion vs React performance on WebShop across 100 customer shopping requests. ReAct + Reflexion fails to significantly outperform ReAct").
This bounds the method's applicability: Reflexion helps where the action/observation space is legible enough for the backbone to reason about (AlfWorld actions visible in observations; Wikipedia search tolerant of imprecise queries), and fails where success needs creative exploration under ambiguous search semantics.