Library Drift: Diagnosing and Fixing a Silent Failure Mode in Self-Evolving LLM Skill Libraries — L2 #

§1 TL;DR #

Self-evolving skill libraries silently degrade via "library drift" — unbounded skill accumulation without outcome-driven lifecycle management causes retrieval dilution and harmful injection. Three governance mechanisms (outcome-driven retirement, bounded active-cap, meta-skill authoring prior) lift held-out pass@1 by +0.328 on hard coding tasks; eight ablations show meta-skill is the single most valuable component and explicit deduplication is unnecessary.

§2 痛点 / 方法 / 结果 #

Q1 — 痛点 #

Self-evolving skill libraries (Voyager, ExpeL, CASCADE, AutoSkill) let frozen LLM agents accumulate reusable procedural knowledge across tasks. The promise is compounding returns: each solved task deposits a skill that accelerates future ones. Reality is different — SkillsBench (Li et al., 2026) shows LLM-authored skills deliver +0.0pp gain over no-skill baselines while human-curated ones deliver +16.2pp. A survey of 20+ such systems reports that lifecycle management (versioning, conflict detection, deprecation) is "largely neglected." The libraries grow, but the agents do not improve.

The root cause is library drift: as skills accumulate without quality gates, the library (1) dilutes retrieval precision with marginal or redundant entries, (2) injects stale or harmful guidance into the solver, and (3) causes performance stagnation or active regression — all without any explicit error signal. The failure is silent, making it indistinguishable from inherent task difficulty variation in end-task metrics alone.

Q2 — 方法 #

The paper provides three contributions organized around diagnosis and cure:

1. Failure definition + reproducible trigger. Library drift is operationally defined (Eq. 1) and bracketed by two ablations: A1 (no skill injection → flat floor at +0.002) and A4 (harsh retirement → active harm at $-0.019$, below the no-skill baseline).
1. Trace-level diagnostics. Three signals expose drift before end-task metrics decline: per-skill contribution score $\hat{c}(s)$, attribution verdicts (helped/hurt/neutral from LLM-as-judge), and router engagement metrics (fraction of tasks assigned a skill vs. none).
1. Verified fix — "Ratchet." A minimal governance recipe with three mechanisms: (a) outcome-driven retirement — retire skill $s$ when $n(s) \geq N_{\min}$ trials and $\hat{c}(s) \leq -\tau$; (b) bounded active-cap $C$ — hard limit on active skills, forcing eviction of lowest-contribution skills; (c) meta-skill authoring prior — constrains the synthesizer to produce stylistically consistent skills, reducing harmful births.

核心技术壁垒: The Hoeffding-calibrated evidence floor ($N_{\min} = 100$) that gates retirement decisions. A4 demonstrates that acting on insufficient evidence ($N_{\min} = 20$, Hoeffding $\epsilon \approx 0.44$) collapses the skill bank to 2 active entries and actively harms performance — worse than no governance at all. The insight is that the evidence floor, not the retirement mechanism per se, is the load-bearing design choice: premature eviction of useful skills on unlucky statistical draws is more damaging than retaining marginal ones.

Q3 — 结果 #

On MBPP+ hard-100 (100 tasks the model cannot reliably solve, 100 rounds, 3 seeds, Claude Opus 4.7):

• Default (full Ratchet): baseline 0.258 → peak 0.658, late-window gain +0.328 ± 0.018 — more than doubling performance on genuinely difficult tasks.
• A1 (no injection): +0.002 — the no-skill floor; skill creation without injection produces zero gain.
• A4 (harsh retirement): $-0.019 \pm 0.010$ — below no-skill baseline; governance on insufficient evidence causes active harm.
• A3 (no meta-skill): +0.187 — removing the authoring prior costs 43% of Default's gain, making it the single most valuable component.
• A5/A6 (no explicit dedup): +0.374 / +0.363 — slightly exceed Default; meta-skill subsumes explicit deduplication.
• A7 (cap=100): +0.317 — comparable mean but variance balloons from ±0.018 to ±0.110; cap primarily controls variance.

§3 架构 / 方法図 #



Paper Figure 1: The Ratchet loop. Inference layer (top): Router → Solver → Grader → Capsule. Memory layer (middle): Skill Bank, Meta-Skill, Evidence Log. Reflection layer (bottom): Critic → Curator. Without retirement and the bounded cap, the Skill Bank accumulates unchecked and library drift emerges.

The architecture operates a five-phase loop per round: (1) Eval — route active skills on held-out tasks; (2) Train — same pipeline on the train split, generating failure substrate; (3) Critic — per-failure attribution verdict via separate LLM call (helped/hurt/neutral/inapplicable + pattern label + confidence); (4) Synthesizer — cluster failures by canonical pattern, author new skills from clusters with $\geq 3$ members; (5) Curator — compute contribution scores $\hat{c}(s)$, retire under-performers ($n(s) \geq 100$ and $\hat{c}(s) \leq -0.10$), enforce active-cap ($C = 50$).

The Router uses two-stage retrieval ($K = 10$ tf-idf + $K = 10$ embedding via Cohere embed-v4) followed by an LLM gate that selects one skill or none. The gate's ability to decline injection is itself a drift-prevention mechanism — A2 (retrieval-only, bypassing the gate) shows 98% engagement but degraded quality (+0.077 vs. Default's +0.328).

stateDiagram-v2 direction LR [*] --> Eval: round t Eval --> Train: held-out scores Train --> Critic: failure capsules Critic --> Synthesizer: attribution verdicts Synthesizer --> Curator: candidate skills Curator --> [*]: updated bank state Curator { ComputeC: compute ĉ(s) Retire: retire if n≥100, ĉ≤-0.10 Cap: enforce |S|≤50 ComputeC --> Retire Retire --> Cap }

Three drift sub-modes emerge depending on governance strength: stagnation (skills never reach the solver — A1), bloat (unbounded growth degrades retrieval — prevented by the cap), and erosion (over-aggressive retirement destroys useful skills faster than they accumulate — A4).

§4 作者证明 #

记号表 #

Symbol	Definition	Introduced
$\mathcal{S}_{t}$	Active skill set at round $t$	§3.1
$p_{0}$	No-skill pass rate (baseline)	§3.1
$\hat{c}(s)$	Empirical contribution score for skill $s$	§4.1
$n(s)$	Number of trials where skill $s$ was injected	§4.1
$N_{\min}$	Evidence floor (minimum trials before retirement decision)	§5.1
$\tau$	Retirement threshold on contribution score	§5.1
$C$	Active-cap (max concurrent active skills)	§5.2
$\epsilon$	Hoeffding estimation tolerance at $N_{\min}$ trials	§5.2
$\delta$	Per-skill failure probability (union bound parameter)	§5.2

方程物理意义 #

Eq. 1 — Drift condition:

$$\mathbb{E}[\text{pass@1}\mid\mathcal{S}_{t}] < \mathbb{E}[p_{0}]$$

The library's expected performance falls below what the agent achieves without any skills. This operationally defines when a skill library has become net-harmful.

Eq. 2 — Per-skill contribution score:

$$\hat{c}(s) = \frac{\text{successes}(s) - \text{failures}(s)}{\text{trials}(s)}$$

Normalised difference between success and failure counts when skill $s$ was injected, measuring net contribution. A declining mean $\hat{c}$ across the bank is an early-warning signal of drift.

Proposition 1 — Non-divergence bound:

$$\mathbb{E}[\text{pass@1}] \geq \mathbb{E}[p_{0}] - (\tau + \epsilon) - C\delta$$

Under bounded cap $C$ and retirement threshold $\tau$, the governed library cannot degrade arbitrarily below the no-skill floor. With Default values ($\tau = 0.10$, $N_{\min} = 100$, $C = 50$, $\delta = 10^{-3}$): $\epsilon \approx 0.20$, floor $= \mathbb{E}[p_{0}] - 0.35$. The bound is loose at this scale (the system gains +0.328 rather than losing 0.35) but rules out unbounded degradation — a guarantee that prior systems (Voyager, ExpeL, AutoManual) without bounded $C$ and $\tau$ cannot provide.

6 项验证检查 #

#	Check	Result
1	Units/dimensions: $\hat{c}(s)$ is dimensionless (ratio of counts), bounded $[-1, 1]$; Prop 1 bound has consistent units (probability)	✓ consistent
2	Limiting case — $C \to 0$: no skills active, $\mathbb{E}[\text{pass@1}] = \mathbb{E}[p_{0}]$; Prop 1 recovers this since $C\delta \to 0$ and no skill is injected	✓ recovers baseline
3	Limiting case — $\tau \to \infty$: all skills immediately retired, equivalent to $C = 0$; system degenerates to no-skill baseline	✓ consistent with A1
4	Ablation correspondence: A4 ($N_{\min} = 20$, $\tau = 0$) violates Prop 1's implicit requirement of sufficient $N_{\min}$ — Hoeffding $\epsilon \approx 0.44$ at $N_{\min} = 20$ vs. $\epsilon \approx 0.20$ at $N_{\min} = 100$; premature retirement confirmed experimentally	✓ theory predicts A4 fragility
5	Error bars: all gains reported as mean ± std over 3 seeds; Default ±0.018, A4 ±0.010 — gain signs consistent across all seeds (Table 3)	✓ significant at $n = 3$
6	Claim-evidence coverage: three claims (drift definition, diagnostics, fix) each have dedicated sections (§3, §4, §5) with experimental validation (§6 ablations A1–A8)	✓ complete coverage

Agent-specific 验证 #

• Failure mode classification: three drift sub-modes (stagnation, bloat, erosion) identified, with each sub-mode reproduced by a specific ablation — stagnation by A1, erosion by A4; bloat prevented by the cap in Default.
• Success-rate model: empirical sweep across 9 conditions × 3 seeds; monotonicity holds along the governance axis (no governance < partial < full), except A4 which demonstrates non-monotonicity at the aggressive end.
• Formal guarantees: Prop 1 provides a probabilistic non-divergence bound — unusual for agent papers. The bound is loose but structurally informative: it shows which knobs ($C$, $\tau$, $N_{\min}$) control worst-case behavior.

§5 实验与数据 #

主结果 #



Paper Table 1: MBPP+ hard-100, mean ± std over 3 seeds. Default achieves +0.328 gain; A1 (no injection) establishes the floor at +0.002; A4 (harsh retirement) falls below baseline at −0.019. A5/A6 (relaxed dedup) slightly exceed Default, demonstrating that meta-skill subsumes explicit filtering.

The Default condition more than doubles pass@1 on genuinely difficult tasks (0.258 → 0.584 late-window mean, peak 0.658). The 73% router engagement indicates the LLM gate is selective — declining injection on ~27% of tasks where no skill is relevant.

学习曲线 #



Paper Figure 2: Held-out pass@1 by round (3-seed mean ± 1 std). A1 (flat) and A4 (declining) exhibit library drift. Default, A5, A6, A8 all converge to similar final performance. A7 (doubled cap) shows comparable mean but substantially wider confidence band.

The learning curves reveal three regimes: (1) fast initial climb (rounds 0–30) as the first useful skills are synthesized; (2) plateau (rounds 30–60) as the bank approaches the cap and retirement begins competing with synthesis; (3) stable high performance (rounds 60–100) as the bank reaches equilibrium. A4 diverges downward after round 20, coinciding with bank collapse to 2 active skills.

消融分解 #

Components that are necessary:

• A1 (no injection): +0.002 — skill creation without injection is useless; the routing + injection pathway is essential.
• A2 (retrieval-only): +0.077 (24% of Default) — the LLM routing gate contributes 76% of the gain beyond raw retrieval.
• A3 (no meta-skill): +0.187 (57% of Default) — the meta-skill authoring prior is the single most valuable component, worth $-0.141$ when removed.

Harsh retirement is harmful:

• A4 ($N_{\min} = 20$, $\tau = 0.0$): $-0.019$ — below no-skill baseline, consistent across all three seeds ($-0.005$, $-0.027$, $-0.025$). At $N_{\min} = 20$, skills with true contribution $c \in [-0.44, 0]$ are prematurely retired on unlucky draws.

Explicit deduplication is unnecessary:

• A5 (no pattern canonicalisation): +0.374; A6 (no cover-guard): +0.363 — both slightly exceed Default (+0.328). The meta-skill's stylistic guidance already prevents enough duplication that the explicit filters' false positives cost more than the duplicates they catch.

Relaxed knobs:

• A7 (cap=100): +0.317 — comparable mean but variance balloons (±0.110 vs. ±0.018). The cap's primary function is variance control.
• A8 (meta-skill refresh every 10 rounds): +0.372 and highest peak (0.725), but 55% more wall time (10.1h vs. 6.5h) — not justified at this scale.

Per-seed 一致性 #



Paper Table 3: Per-seed rolling-mean gain. Default is consistent across seeds (0.303–0.343). A4 is consistently negative (−0.005, −0.027, −0.025). A7's high variance is visible in the per-seed spread (0.172–0.440).

运行成本 #

Default: ~14.5k LLM calls per 100 rounds (10k solver + 4.3k critic + 152 synthesis), 43% more than the no-skill baseline. Wall time: 6.5h (2.8× A1's 2.3h). Solver dominates cost at 63–99% of total calls across conditions.

§6 论证链 #

Step	Claim	Evidence	Link
1	LLM-authored skill libraries fail to improve agents	SkillsBench: +0.0pp (LLM) vs. +16.2pp (human); survey of 20+ systems reports lifecycle management "largely neglected"	§1
2	The root cause is library drift: unbounded accumulation degrades performance below the no-skill baseline	Eq. 1 defines drift operationally; mechanism decomposed into 3 stages (accumulation → retrieval degradation → silent injection harm)	§3.1–3.2
3	Drift is reproducibly triggered by ablation	A1 (no injection) establishes the no-skill floor at +0.002; A4 (harsh retirement) triggers active harm at $-0.019$	§3.3
4	Trace-level diagnostics detect drift before end-task metrics decline	Per-skill $\hat{c}$, attribution verdicts, and router engagement expose A4's bank collapse at round 30 — before pass@1 shows clear decline	§4
5	Three governance mechanisms jointly fix drift	Retirement + cap + meta-skill → +0.328 gain; each mechanism targets one drift stage; Prop 1 guarantees finite worst-case degradation	§5
6	Meta-skill is the single most valuable component; explicit dedup is subsumed	A3 (no meta-skill) loses 43% of gain; A5/A6 (no dedup) slightly exceed Default — meta-skill's stylistic consistency prevents enough duplication that explicit filters' false positives cost more than their catches	§6.3
7	Governance is not uniformly beneficial — evidence floor is load-bearing	A4 with $N_{\min} = 20$ (Hoeffding $\epsilon \approx 0.44$) collapses bank to 2 skills and harms performance; the calibrated evidence floor ($N_{\min} = 100$, $\epsilon \approx 0.20$) is what makes retirement safe	§3.3, §6.3

§7 实现 cross-reference #

[实现未公开]

核心技术壁垒 #

The Hoeffding-calibrated evidence floor ($N_{\min} = 100$) that separates safe governance from harmful governance. The paper's most counterintuitive result is A4: reducing $N_{\min}$ from 100 to 20 and tightening $\tau$ to 0.0 — ostensibly more aggressive quality control — produces the worst outcome of all conditions. At $N_{\min} = 20$, the Hoeffding tolerance is $\epsilon \approx 0.44$, meaning skills with true contribution anywhere in $[-0.44, 0]$ face substantial probability of premature retirement on unlucky draws. The resulting bank collapse to 2 active skills leaves the router with near-nothing to select — yet the surviving skills are the ones that happened to get lucky early, not necessarily the best. Replicating this calibration requires understanding that the evidence floor and the retirement threshold interact through the Hoeffding bound, and that the floor must be set large enough to make the tolerance $\epsilon$ smaller than the threshold $\tau$.

关键実装細節 #

1. LLM routing gate as drift prevention: A2 shows retrieval-only routing (bypassing the LLM gate) achieves 98% engagement but only 24% of Default's gain. The gate's ability to select "none" — declining to inject any skill — prevents low-quality injections and is itself a passive drift-prevention mechanism. Systems that always inject the top retrieval hit lack this safety valve.
1. Meta-skill subsumes deduplication at moderate scale: A5/A6 demonstrate that explicit pattern canonicalisation and cover-guard filters are not just unnecessary but mildly counterproductive at 100-task scale — their false positives discard useful skills that the meta-skill's authoring guidance would have made sufficiently distinct. The paper cautions this may reverse at larger scale where the meta-skill's implicit consistency pressure becomes insufficient.