Remember When It Matters: Proactive Memory Agent for Long-Horizon Agents — L2 #

1. TL;DR #

A separate memory agent runs beside an unmodified action agent, maintaining a structured memory bank every N steps and deciding whether to inject a grounded reminder or stay silent. Fixes "behavioral state decay". +8.3 pp Terminal-Bench, +6.8 pp τ² (Sonnet 4.5); learnable on Qwen3.5-27B.

2. Q1 / Q2 / Q3 #

Q1 — 痛点 (the problem) #

Long-horizon agents (command-line automation, multi-turn tool use) fail not because they cannot solve each local step, but because decision-relevant execution state stops controlling behavior as the trajectory grows. The paper names this behavioral state decay: task requirements, environment facts, failed attempts, failure diagnoses, and open subgoals may still sit in the transcript (or even inside the context window) yet no longer reliably shape the next action. Three concrete symptoms: an agent identifies a requirement early then violates it while fixing an unrelated bug; retries a near-identical command that already failed; or re-diagnoses an error pattern it already characterized. Crucially, the paper argues this is not a storage/retrieval problem — "simply making longer histories available is insufficient" — and it is a stronger control question than summarization: a summarizer asks what to retain; this work asks whether any retained state should become active in the next decision.

Scope classification (agent-specific):

• Task class: open-ended autonomous (Terminal-Bench command-line) + semi-closed interactive tool-use with domain rules (τ²-Bench airline/retail/telecom). Not single-turn RAG.
• Interaction pattern: multi-turn, long-horizon (many tool-use rounds per task).
• Autonomy: fully autonomous action agent (Terminal-Bench) and user-simulator-coordinated (τ²); the memory agent is a fully-autonomous supervisory process, no human in the loop.

Q2 — 方法 (the method) #

Introduce a memory agent $\pi_M$ that runs alongside an unchanged action agent $\pi_A$. At a fixed interval (every step, window $k=8$), the memory agent executes a strictly ordered two-phase workflow. Phase 1 (memory management) issues a list of predefined tool calls — memory_update_status, memory_save_knowledge, memory_save_procedural, memory_delete — that the system executes in order to produce the updated bank; it never free-form rewrites. Phase 2 (intervention selection) reads the updated bank and emits either a targeted reminder or . A non-null reminder is injected as transient memory context into only the next action-agent call; the action agent's base instructions, tools, and decoding are untouched.

核心技术壁垒 (THE single hardest-to-replicate insight): it is not the memory bank, nor the reminder text — it is the calibrated silence action. Treating "do not intervene" as an explicit, first-class action (rather than intervening whenever memory updates) is what separates this from advisor models, always-on injection, and retrieval. The bank+reminder machinery is straightforward to copy; what is hard is the intervention-timing policy that decides when remembered state is worth entering the control loop without adding distraction, tokens, or latency. This is precisely the axis the paper later trains with GRPO because prompting calibrates it imperfectly.

Q3 — 结果 (the outcome) #

Plug-and-play memory improves pass@1 for both weaker and stronger action agents on two benchmarks: Terminal-Bench 2.0 Sonnet 4.5 37.6% → 45.9% (+8.3 pp), Opus 4.6 43.5% → 45.9% (+2.4 pp); τ²-Bench Sonnet 55.0% → 61.8% (+6.8 pp), Opus 66.2% → 68.7% (+2.5 pp). Gains are larger for the weaker agent but do not vanish for the stronger one. Ablations argue that selective intervention beats passive full-bank exposure, always-on injection, advisor-only guidance, and general (Mem0) retrieval. As a preliminary result, the intervention policy is partly learnable on open-weight Qwen3.5-27B (SFT then GRPO), transferring to held-out Terminal-Bench (+3.5 pp).

3. 架构 / 方法图 #

Paper's Figure 1, verbatim (caption: "Overview of the memory-intervention architecture. (a) The memory agent runs as a separate process beside an unmodified action agent. (b) Within each memory step, a two-phase workflow first updates the structured memory bank, then decides whether any remembered state should enter the action agent's next decision.").

Panel (a) shows the decoupling that makes the method plug-and-play: the action agent drives the environment loop while the memory agent observes a sliding window of recent steps plus the current bank and, at every $N$ steps, may inject a reminder into the next action call. Panel (b) is the load-bearing detail — Phase 1 edits the bank via tool calls (status / knowledge / procedural), and only then does Phase 2 read the updated bank to choose reminder-vs-silence. The reader should notice that Phase 2 is conditioned on $B_t$ (post-edit), never on the pre-edit bank, and that the private status field never reaches the action agent.

The agent loop (one memory step) #

Planning & reasoning #

• Planning style: not tree-search; a fixed-schedule supervisory loop layered on top of the action agent's own ReAct-style tool-use. The memory agent itself is a two-phase prompted workflow, not a planner.
• Decomposition: neither top-down subtask decomposition nor branch-and-fold; the memory agent maintains flat structured state (status/knowledge/procedural) and reactivates fragments of it.
• Budget: memory agent invoked at step 1 then every step; recent window $k=8$ messages. No max-step cap reported for the action agent beyond the harness defaults.
• Backtracking: the action agent cannot undo actions via memory; the memory agent can only memory_delete stale/incorrect entries — memory is editable, environment actions are not reversible through this mechanism.

Memory model. Short-term = action agent's own context window (where decay happens). Long-term = the memory agent's structured bank $B_t=(s_t,K_t,P_t)$: private status, stable knowledge facts, procedural attempt/outcome records. Episodic = the sliding trajectory window $w_t$ the memory agent re-reads each step. The bank is deliberately outside the action context so it does not pollute the action agent until Phase 2 chooses to surface a fragment.

4. 作者证明 #

无形式化作者证明 — 仅实证. The paper provides a formal setup (Eqs 1–2) but no convergence, regret, or success guarantee — as is typical for agent papers. What could have been bounded but was not: the marginal success probability of an intervention vs silence (an intervention-utility bound), or a token/latency-vs-gain trade-off frontier. The claims rest entirely on pass@1 sweeps.

Notation table #

Equations and physical meaning #

The edit-then-intervene update is strictly sequential — the bank is edited first, then the intervention is conditioned on the updated bank $B_t$ (not $B_{t-1}$):

$$B_t \sim \pi_M^{\mathrm{edit}}(\cdot \mid x, w_t, B_{t-1}), \quad i_t \sim \pi_M^{\mathrm{intervene}}(\cdot \mid x, w_t, B_t)$$

The bank is a triple separating the agent's private working state from stable facts from procedural evidence:

$$B_t=(s_t,K_t,P_t)$$

Physical meaning: decoupling maintenance ($\pi_M^{\mathrm{edit}}$) from action selection ($\pi_M^{\mathrm{intervene}}$) is what lets "silence" be a well-defined action — Phase 2 evaluates a stable, freshly-edited state rather than reacting to raw trajectory.

6 minimum checks #

1. Dimensional/type check — $i_t$ has exactly two branches ($\varnothing$ or reminder); the null branch is real (Table 2 "Always inject" removes it and loses macro/airline). Consistent.
2. Ordering check — Phase 2 reads $B_t$ not $B_{t-1}$ (Eq 1). L1 §3.3 confirms Phase 2 "does not modify the memory bank" and is "conditioned on the updated memory bank." Consistent.
3. Monotonicity vs backbone strength — gain shrinks as action agent strengthens (Sonnet +8.3 → Opus +2.4 on Terminal-Bench; +6.8 → +2.5 on τ²). Monotone-decreasing but strictly positive; matches "not merely compensating for limited capacity."
4. Monotonicity vs training stage — SETA reward: base-27B 0.693 (hurts) → SFT 0.720 → GRPO 0.734. Monotone-increasing after the untrained dip. Consistent with "learnable but requires calibration."
5. Ablation additivity — removing Phase 2 (Full-bank) costs 2.8 macro; removing the bank (Injection-only) is unstable (airline 62.0 < 68.0 baseline). Both components load-bearing; not strictly additive since Injection-only wins telecom.
6. Arithmetic self-consistency — fails softly: §4.2 text states retail Sonnet lift "+9.7 pp" but Table 1 gives 49.1→58.8 = +9.6 pp; and Table 1 (retail 58.8 / avg 61.8) disagrees with Table 2 (retail 57.0 / micro 61.2) for the same "Sonnet + full memory" config. Discrepancies preserved, not resolved by the paper.

5. 实验与数据 #



Paper's Table 1 (caption: "Main results on Terminal-Bench and τ²-Bench. All scores are pass@1 percentages; Δ is reported in percentage points (pp). The memory agent is Claude Opus 4.6."). Load-bearing evidence for the headline claim: memory helps both action-agent strengths on both benchmarks. Notice the τ² per-domain spread for Sonnet — airline +10.0 and retail +9.6 pp are large, telecom only +2.6 pp — which the authors read as evidence that memory is a domain-sensitive intervention policy, not a uniform summarizer.



Paper's Table 2 (caption: "Ablations on τ²-Bench with Sonnet 4.5 as the action agent, Opus 4.6 as the memory agent... Macro averages domains equally, micro is task-weighted."). This is the argument's core: Full-bank context, which keeps only the bank-management phase, trails the full system by 2.8 macro / 2.6 micro points; Always-inject removes silence and slips on macro/airline; Injection-only (no bank) hurts airline (62.0 vs 68.0); Mem0 retrieval wins retail (59.6) but loses airline and macro. The reader should notice no single variant dominates every domain — the full system's claim is "most balanced macro-average," and the margin over Always-inject on micro is a mere −0.3 (i.e., Always-inject slightly leads micro).



Paper's Table 3 (caption: "Representative mechanisms observed in qualitative analysis. Memory interventions help when they reactivate execution state that the action agent previously saw but no longer uses in its next decision."). Five recurring mechanisms — requirement/policy reactivation, environment grounding, failure-loop avoidance, diagnostic carryover, progress/entity tracking. This grounds the "specific, grounded, timely" story (e.g. the airline Gold-vs-Regular case where memory reminded the agent to trust verified records over the user's claim).



Paper's Table 4 (caption: "Training and transfer of the open-weight memory agent. The action agent is a frozen Qwen3.5-122B-A10B; the trainable memory agent is Qwen3.5-27B..."). Panel (a) SETA validation: untrained 27B memory hurts (0.709→0.693), SFT recovers (0.720), GRPO improves (0.734). Panel (b): SETA-trained memory transfers to held-out Terminal-Bench, 37.6% → 41.1% (+3.5 pp). The key takeaway is that memory is net-negative until calibrated — the value lives in the intervention-timing policy, not the memory machinery.

Evaluation summary (agent-specific):

• Benchmarks: Terminal-Bench 2.0 (85 paired tasks, 4 docker failures excluded) and τ²-Bench (airline 50 / retail 114 / telecom 114 = 278 tasks).
• Metric: pass@1 only (binary verifier pass fraction for TB; task-evaluator pass for τ²). No pass@k, no cost-adjusted metric.
• Baselines: the frontier action agent alone; plus ablation baselines standing in for passive augmentation, always-on injection, advisor-style (asawa2025advisor / anthropic2026advisor), and RAG-style retrieval (Mem0).

6. 论证链 #

Step	Claim	Paper-internal support
1	Long-horizon agents fail because decision-relevant state stops controlling behavior even when still present (behavioral state decay).	§1 three symptom examples; framed as distinct from storage/retrieval.
2	Therefore the useful question is when to reactivate state, a stronger control problem than summarization.	§1/§2 positioning; memory reframed as an intervention policy over $i_t$.
3	A separate two-phase memory agent (edit bank, then decide reminder-vs-silence) operationalizes this without touching the action agent.	§3 Eqs 1–2, Fig 1, tool-call interface.
4	This architecture raises pass@1 for both weak and strong action agents on two heterogeneous benchmarks.	Table 1 (+8.3/+2.4 TB; +6.8/+2.5 τ²).
5	Each component is necessary: removing Phase 2 (selection), the silence action, or the bank each weakens robustness differently.	Table 2 ablations (Full-bank −2.8 macro; Always-inject loses macro; Injection-only hurts airline).
6	The intervention timing policy — not the bank — is the hard, learnable part; it can be distilled into an open-weight model.	§4.5 Table 4: untrained hurts, SFT recovers, GRPO improves, +3.5 pp transfer.

7. 实现 cross-reference #

Public code: https://github.com/yifannnwu/proactive-memory-agent (referenced in L1 header). The L1 does not include file:line extracts from that repository, so concrete symbol locations are [实现未公开] at the L1 layer; the following are grounded in the paper's own interface description.

• Tool catalog & schema (§3.3): four Phase-1 tools — memory_update_status, memory_save_knowledge, memory_save_procedural, memory_delete. Calls are returned as a list and executed in order; no free-form bank rewrite. Argument schema is entry-oriented (short identifier + natural-language content + metadata: creation time, access statistics).
• Side effects: memory tools are the only writes and are reversible (memory_delete by identifier); the environment/tool actions of the action agent are outside this mechanism and are not undoable through memory.
• Error surface / environment contract: the paper does not specify how a broken action-agent tool is stringified back to the memory agent; the memory agent only sees the trajectory window $w_t$. Environment is treated as the benchmark harness's containerized (Terminal-Bench, Docker) or user-simulator (τ²) contract.

核心技术壁垒 (dedicated paragraph). The genuinely hard-to-replicate element is the selective-silence intervention policy. Anyone can maintain a structured bank and synthesize a reminder; the paper's own ablations show that doing so without calibrated silence (Always-inject) or without grounding (Injection-only) is competitive-at-best or actively harmful in some domains, and that an uncalibrated open-weight memory agent is net-negative before training. The moat is the decision boundary between "reactivate this fragment now" and "stay silent," which the paper only reaches reliably via GRPO on pivot turns — and even the prompted Opus version leaves residual calibration errors (over-confident speculation, redundant reminders, unnecessary verification) as the dominant remaining failure class.

关键实现细节 (easy-to-miss tricks).

1. Phase 2 conditions on the post-edit bank $B_t$ (Eq 1), and the private status field $s_t$ is never shown to the action agent — the memory agent keeps a working task model without polluting action context. Missing either detail changes the method.
2. GRPO focuses updates on "pivot turns" identified by labeled offline rollouts (§4.5), because task-level verifier rewards are too sparse across many memory calls to credit-assign directly — a reward-shaping trick that makes the sparse long-horizon signal trainable.

LLM backbone requirements (agent-specific §7) #

• Memory-agent backbone: default Claude Opus 4.6 — a frontier model equal to or stronger than the action agents it assists, which is the acknowledged cost caveat motivating the open-weight direction.
• Minimum size: Qwen3.5-27B is shown workable after SFT+GRPO; the untrained 27B hurts, so raw small backbones are insufficient without calibration.
• Required capabilities: reliable structured tool-call output for bank management and disciplined structured-output tags / for intervention selection.
• Serving cost: adds one memory-agent (frontier) call at every memory step; the paper flags this cost only for the open-weight direction and does not report tokens-per-task.