AutoMem: Automated Learning of Memory as a Cognitive Skill — L2 #

1. TL;DR #

Treat an LLM agent's file-system memory as a trainable skill, not a fixed module. Two meta-LLM outer loops — one revising the memory scaffold (code/prompts/schema), one LoRA-training a memory specialist on the agent's own good decisions — lift a frozen 32B agent 2×–4× on Crafter/MiniHack/NetHack, reaching frontier-proprietary level.

2. Q1 / Q2 / Q3 (痛点 / 方法 / 结果) #

Q1 — 痛点. Long-horizon agent tasks (up to $10^{4}$–$10^{5}$ steps) overflow the context window, so external memory is needed. But prior external-memory designs bake in a fixed mechanism (RAG, MemGPT paging, memory streams). Worse, memory quality resists manual tuning: a memory mistake at step 50 (a missing map coordinate, a duplicate entry) may only surface as a failure at step 800, and no human can review a $10^{5}$-step trace to find it. The reward-signal view (final progression) discards exactly the trajectory structure that reveals where memory went wrong.

Q2 — 方法. Reframe memory management as an independently learnable skill decomposed into two axes: structure (the scaffold: prompts, file schema, action vocabulary) and proficiency (the model's parametric ability to choose well). File-system ops (read/write/search/append/create, plus special tokens like <|APPEND|>/<|SEARCH|>) become first-class actions sharing the same action space as world actions, making every memory decision a traceable action. AutoMem automates both axes with two sequential outer loops sharing one inner-loop agent:

• Outer-loop #1 (structure): a meta-LLM reads full episode traces as a "code reviewer", diagnoses memory failures, and rewrites the scaffold — kept only if measured progression improves (gated).
• Outer-loop #2 (proficiency): a meta-LLM "training engine" filters the agent's own verbatim memory decisions into supervised data and jointly picks the LoRA config, training a dedicated memory specialist while the gameplay model stays frozen.

核心技术壁垒: using a strong meta-LLM as a trajectory-level code reviewer over complete $10^{5}$-step traces to localize delayed-consequence memory failures — replacing scalar reward with a structured diagnosis that both revises code and selects training data. (See §7.)

Q3 — 结果. With frozen Qwen2.5-32B-Instruct: scaffold opt alone yields Crafter ×1.89 (25.0→47.27%), MiniHack ×3.67 (7.5→27.5%), NetHack ×3.74 (0.42→1.57%). Proficiency training stacks a further ~9–18% relative lift (Crafter→51.36, MiniHack→30.0, NetHack→1.85). The optimized 32B beats Qwen2.5-72B on all three games and matches Claude-Opus-4.5 / Gemini-3.1-Pro-Thinking.

3. 架构 / 方法图 #

Paper's Figure 3, verbatim (caption: "Overview of AutoMem. Two automated outer loops optimize a shared inner-loop agent that uses the file system as its memory. Outer-loop #1 (top): a meta-LLM reviews full episode traces and iteratively revises the agent scaffold. Outer-loop #2 (bottom): a meta-LLM training engine jointly orchestrates data curation and finetuning configuration to train a dedicated memory specialist ... while the task model (frozen, unmodified) commits task actions.").

The shared inner-loop agent (gray center) runs a two-phase per-step loop: a LOG routine ("what is worth recording?") and a PLAN routine ("what do I need to recall to act now?"). Loop #1 sits above it revising the scaffold; loop #2 sits below it training the specialist inside the converged scaffold. The reader should notice the two loops are sequential: #1 sets the structural ceiling, #2 pushes the model toward it.

The Mermaid adds the per-step control flow of the two-model shared-conversation inference (lower panel of Fig 3) that the raster overview compresses: both instances take turns on one history, with a handoff after the specialist's last memory op.

4. 作者证明 #

无形式化作者证明 — 仅实证. The paper has 0 numbered display equations, no convergence theorem, no variance/sample-complexity bound. The only formalization is an ML-metaphor in §2.3 "Two loops, one objective": each loop has parameters $\theta$ and an update signal $\nabla L$ from meta-LLM trajectory analysis — loop 1: $\theta$ = scaffold, $\nabla L$ = a code revision; loop 2: $\theta$ = memory-model weights, $\nabla L$ = a matched (curated-data, LoRA-config) supervised step. This is analogy, not a literal gradient.

What a formal guarantee would have looked like: a monotone-improvement statement for the gated loop #1 (progression is non-decreasing across accepted revisions by construction of the accept-if-strictly-better gate), and a generalization bound for the tiny LoRA sets (444–1597 examples) relating specialist proficiency to trace-pool coverage. Neither is proven; loop #1's monotonicity holds only on the fixed eval seeds, not the seed distribution.

Six minimum sanity checks against the empirical claims:

1. Scaffold multipliers self-consistent. 47.27/25.0 = 1.89, 27.5/7.5 = 3.67, 1.57/0.42 = 3.74 — match the reported ×.
2. Weights frozen during loop #1. Only code/prompts/schema change; gains cannot be attributed to model updates. ✓ internally consistent.
3. Train/eval separation. Training seeds explicitly disjoint from eval seeds $[42..51]$ (Appendix A.2) → no seed contamination.
4. Gated acceptance. A revision kept only if avg progression strictly exceeds prior iter → measured gains cannot regress on eval seeds.
5. Stacking, not trading off. Gameplay weights untouched in loop #2, so the +train lift adds on top of scaffold rather than degrading task-format competence. ✓
6. Filter-not-teacher. Every training example is verbatim base-model output; no distillation from Opus into the data → the specialist's gain is genuine self-improvement, not covert teacher distillation. ✓

5. 实验与数据 #

Method core diff (before vs after).

	Before (v0 baseline)	After (AutoMem)
Memory	fixed module (sliding window / append-only files)	first-class file-system actions in shared action space
Structure tuning	hand-designed, static	meta-LLM code-reviewer revises scaffold, gated on progression
Model proficiency	untuned base	LoRA memory specialist trained on own good decisions
Optimization signal	scalar final return	full-trace trajectory review ($10^{5}$ steps)



Paper's Table 1 (progression rate %, mean ± se). Scaffold opt = loop #1 at convergence (Crafter v5 / MiniHack v4 / NetHack v2); + memory training adds loop #2. The load-bearing comparison: the AutoMem + memory training row (51.36 / 30.00 / 1.85) sits beside Qwen2.5-72B (27.3 / 5.0 / 0.3) and Claude-Opus-4.5 (49.5 / 27.5 / 2.0). Notice memory optimization on 32B beats 72B by a wide margin — memory structure outweighs 2.25× parameter scale. It still trails the top frontier proprietary rows (Gemini-3-Pro Crafter 57.3, MiniHack 40.0; all four proprietary on NetHack).



Paper's Figure 1: progression rate vs scaffold version (v0→v5/v4/v2) then +train, three curves. Shows loop #1 converges in only 2–5 iterations per environment, and +train adds a clean final step comparable in magnitude to one-or-two scaffold iterations. The reader should notice the curves are near-monotone — a direct consequence of the accept-if-better gate.



Paper's Figure 4 (v0 vs final scaffold; lower is better everywhere). Four behavioral indicators quantify why progression rises: unproductive game-action rate drops 32–65% (left); redundant writes drop −68 to −83%; empty-search rate falls −13 to −50%; per-step input context shrinks −3 to −30%. Critically the model weights are untouched here, yet task behavior improves — evidence for the shared-action-space spillover claim.



Paper's Figure 5: v0 append-only dungeon_map.txt → coordinate-keyed <|UPSERT_MAP|> dedup + auto-synced inventory/status + pre-populated strategy. This single schema change cuts NetHack per-step memory growth from 138 to 6 characters (−95%), and is credited with the jump from dying at Xp 1 to surviving to Xp 2. It concretizes what the aggregate −83% redundant-write number means mechanically.



Paper's Table 2: memory writes per SEARCH, base vs +trained specialist (lower = more consult-before-write). Crafter 0.84→0.39 (−54%), MiniHack 2.89→0.82 (−72%), NetHack 4.66→1.31 (−72%). This shows the specialist internalized the consult-before-write discipline into weights — the same behavior the scaffold prompts for is now a learned policy, not just a prompted one.



Paper's Figure 6: per-environment episode at v0 / evolved scaffold / +trained specialist. Crafter 9→55→59% (wood-only → stone tools + furnace + iron → also self-feeds); MiniHack Corridor-R3 0→0→100% (specialist solves what neither prior stage could); NetHack Xp 1→2→4. This makes the delayed-consequence thesis concrete: the coordinate-keyed map (Fig 5) is what lets NetHack survive past the base agent's few-hundred-step death.

Training recipe & scale (loop #2).

Stage	Purpose	Data	Config decided by
(a) collection	gather traces	base plays 100 Crafter / 400 MiniHack / 50 NetHack eps, seeds disjoint from eval	fixed
(b) data engine	select + compose	1597 / 444 / 800 final examples, memory-op-only (action part trimmed)	Claude Opus 4.7 --effort max
(c) LoRA	absorb data	see per-env config below	joint with (b)

Disclosed: LoRA cutoff_len=16384, bf16, AdamW, cosine LR, warmup_ratio=0.05, 2 GPUs, DeepSpeed ZeRO-3. Per-env: Crafter (rank 256, α 512, bs 32, lr 5e-5, 4 ep, attention-only), MiniHack (rank 128, α 256, bs 16, 3 ep), NetHack (rank 256, α 512, bs 32, 1 ep). GPU hours / MFU: [论文未披露].

Convergence & stability. Loop #1 converges in 2–5 iterations (Fig 1). Loop #1 stability comes from the gate + up-to-1 retry with the failed eval log, then clean-session restart (Appendix A.2). For loop #2, the paper's own stability concern is that under a modest LoRA adapter a config suited to one dataset can under/overfit another — hence data and config are chosen jointly per environment. No KL/length-explosion signals apply (this is filtered SFT, not RL).

Dataset analysis. Composition: 100% the agent's own verbatim memory-operation responses (no external/human/web data). Quality filter: deterministic postprocessing strips code-block wrappers, drops examples with only action-commitment (no memory-op signal), and trims the gameplay-action tail from mixed examples. Contamination: train seeds explicitly disjoint from eval seeds $[42..51]$. No synthetic generator, no human annotation.

6. 论证链 #

#	Step	Support (paper-internal)
1	Context window is fixed working memory; long-horizon tasks overflow it → external memory needed.	§1 para 2
2	Putting file-system ops in the shared action space makes every memory decision a traceable action, so it is observable and optimizable.	§2.1 para 2
3	Memory consequences are delayed, so the optimization signal must be trajectory-level, not scalar return; a strong meta-LLM can review full traces like a code reviewer.	§2.2 para 2–3
4	Loop #1 revises the scaffold on that signal, gated on measured progression → 2×–4× gains with frozen weights.	§3.2, Table 1, Fig 4
5	After the scaffold is lean, the remaining bottleneck is parametric proficiency; loop #2 LoRA-trains a specialist on the agent's own good decisions.	§2.3, §3.2
6	Because memory is a separable skill, training the specialist while freezing the gameplay model makes the proficiency gain stack cleanly on the scaffold gain.	§2.3 para 5, Table 1

7. 实现 cross-reference #

Code is open: https://github.com/autoLearnMem/AutoMem (project page https://autolearnmem.github.io/); full prompt templates for both loops released there (Appendix A intro). File:line citations [实现未公开 at line granularity] — the L1 records the repo but not line anchors.

核心技术壁垒 (deep dive). The hardest-to-replicate insight is using the meta-LLM as a trajectory-level reviewer over complete $10^{5}$-step traces rather than a reward function. A scalar return says "this episode scored 1.57%"; it cannot say "at step 50 you failed to dedup the map, which buried the coordinate you needed at step 800." The meta-LLM is given per-step logs + the resulting memory directories + the agent code, and localizes the causal memory fault — then in loop #1 emits a code revision (e.g. the <|UPSERT_MAP|> operation) and in loop #2 emits data-selection criteria derived from what the agent code requires of the model. Replicating this needs a meta-LLM strong enough to read $10^{5}$-step logs coherently (they use Opus 4.6/4.7 --effort max, a newer model than the Opus 4.5 baseline the 32B is said to match), plus the gate/retry harness. Without a sufficiently capable reviewer, the whole diagnose-and-revise decomposition collapses back to scalar-reward search.

关键实现细节 (easy-to-miss tricks).

1. Two-model shared-conversation inference. The LoRA specialist and the frozen base take turns on one conversation history; after the specialist's last memory op a short handoff passes control to the gameplay model, which may still issue further memory reads before committing the world action (§2.3 para 4). Missing the handoff would leak action-format tokens into the specialist's loss.
2. Action-tail trimming in data prep. For examples containing both a memory op and an action commitment, the gameplay-action part is trimmed so the specialist's supervised loss targets only memory-operation reasoning — this is why the tiny LoRA sets teach memory discipline without diluting into action formatting (Appendix A.2).

9. Reproducibility & ecosystem #

Training code is open (repo above); the LoRA recipe uses stock DeepSpeed ZeRO-3 + AdamW, so the closest open reference stack is standard PEFT/LoRA (Hu et al., 2022) — no bespoke trainer. The evaluation runs on the released BALROG harness (Paglieri et al., 2024) with minor documented config changes (Appendix A.1). No community re-implementation reports exist yet (paper dated 2026-07). Architecturally the trained-specialist-alongside-frozen-base topology is closest to MeMo (Quek et al., 2026); the "memory as action" framing overlaps MemAct (Zhang et al., 2025b, on context window) and the scaffold loop overlaps MemEvolve / EvolveMem, but AutoMem is distinguished by optimizing both axes on full long-horizon traces.