Dynamic Skill Lifecycle Management for Agentic Reinforcement Learning #

§1 TL;DR #

SLIM treats the active external skill set as a dynamic optimization variable during agentic RL, using leave-one-skill-out marginal contribution estimation to retain, retire, or expand skills — yielding non-monotonic skill trajectories that outperform both persistent-accumulation and forced-zero-skill paradigms by 7–13 points on ALFWorld/SearchQA.

§2 痛点 / 方法 / 结果 #

Q1 痛点 #

Existing skill-based agentic RL methods follow one of two monotonic paradigms: (1) persistent accumulation (skills only grow, causing routing noise and context bloat), or (2) forced internalization toward zero-skill inference (risks catastrophic forgetting of narrow/long-tail capabilities). Neither accounts for the fact that under finite parametric capacity, the optimal active skill set is non-monotonic, task-dependent, and stage-dependent.

Paper's Figure 1, verbatim. Shows the contrast between monotonic accumulation (SkillRL), forced elimination (Skill0), and SLIM's non-monotonic active skill trajectory over training.

Figure 1 visually establishes the core thesis: external skill count should rise, plateau, and partially decline as the policy absorbs some capabilities while retaining others that resist internalization.

Q2 方法 #

SLIM (Skill LIfecycle Management) decomposes the intractable capacity-constrained allocation problem (Eq. 2) into three tractable components:

1. Hierarchical skill retrieval — task-conditioned embedding similarity filters the combinatorial skill space to a small candidate set ($K \leq 3$).
2. Marginal External Contribution (MEC) estimation — leave-one-skill-out validation measures the performance drop $\Delta_t(s)$ when skill $s$ is removed, smoothed by EMA.
3. Lifecycle operations — retain (MEC ≥ $\tau_{\mathrm{keep}}$), retire (MEC < $\tau_{\mathrm{retire}}$ with patience), expand (persistent failures on routed tasks with low MEC).

These alternate with GRPO-based policy updates: policy improves with fixed active set, then lifecycle operations update the active set with fixed policy.

核心技术壁垒: The leave-one-skill-out MEC estimator is the single hardest-to-replicate insight. It provides a local counterfactual signal that distinguishes "internalized" skills (low MEC because the policy absorbed the capability) from "still necessary" skills (high MEC despite training), enabling contribution-aware lifecycle decisions rather than heuristic scheduling.

Q3 结果 #

• ALFWorld: SLIM† achieves 87.5% (+12.5 over SkillRL†, +13.3 over Skill0)
• SearchQA: SLIM achieves 41.0% (+1.7 over Skill0, +2.9 over SkillRL†)
• Active skill set converges to 21 (from 38 initial + 16 expanded, 33 retired)
• No-skill policy reaches 84.4% (policy improves simultaneously)
• Bootstrap 95% CIs on all gaps exclude zero
• Robust to noisy initialization: 85.6% even from 60% corrupted skill bank

§3 架构 / 方法图 #



Paper's Figure 2, verbatim (caption: "An overview of SLIM. Motivated by Eq. (2), SLIM first retrieves task-conditioned visible skills, then estimates skill-level marginal contribution via leave-one-skill-out validation, and finally updates the policy and skill lifecycle through GRPO-based retain–retire–expand operations.").

The architecture shows the alternating optimization loop: rollouts with skill-conditioned policy feed into GRPO for policy updates, while periodic audit intervals trigger MEC estimation via leave-one-skill-out validation on held-out tasks. The three lifecycle operators (retain/retire/expand) act on the active set before the next training epoch.

stateDiagram-v2 [*] --> SampleTasks SampleTasks --> HierarchicalRetrieval: task x, type k HierarchicalRetrieval --> SkillConditionedRollout: Q_t(x) = TopK filtered SkillConditionedRollout --> GRPOUpdate: trajectories + rewards GRPOUpdate --> AuditCheck: θ updated state AuditCheck <> AuditCheck --> SampleTasks: not audit step AuditCheck --> MECEstimation: every d=10 steps MECEstimation --> LifecycleOps: Δ_t(s) computed state LifecycleOps { [*] --> Retain: MEC ≥ τ_keep [*] --> Retire: MEC < τ_retire, patience met [*] --> Expand: persistent failures + low MEC } LifecycleOps --> SampleTasks: A_{t+1} updated

Memory model: Short-term = context window (skill text injected as prompt prefix); long-term = external skill bank (hierarchical, persistent between episodes); episodic = lifecycle logs tracking MEC history per skill.

Error recovery: When a skill-conditioned rollout fails, the failure is bucketed by task type. Accumulated failures trigger expansion (Eq. 7) rather than immediate retry. Retired skills remain in the inactive bank and can theoretically be re-expanded.

§4 作者证明 #

记号表 #

Symbol	Meaning
$\pi_{\theta}$	LLM agent policy parameterized by $\theta$
$\mathcal{A}_t$	Active external skill set at step $t$
$\mathcal{I}$	Latent internalized skill set
$\mathcal{S}$	Full skill bank
$\Omega(\mathcal{A})$	External support cost (monotone set function)
$\mathcal{C}_{\theta}$	Finite parametric knowledge capacity
$\Delta_t(s)$	Marginal External Contribution of skill $s$
$\bar{\Delta}_t(s)$	EMA-smoothed MEC
$\mathcal{Q}_t(x)$	Task-conditioned retrieved skill subset
$\tau_{\mathrm{keep}}, \tau_{\mathrm{retire}}, \tau_{\mathrm{expand}}$	Lifecycle thresholds

核心方程物理意义 #

Capacity-constrained allocation (Eq. 2):

$$\max_{\theta, \mathcal{A}, \mathcal{I}} \; \mathbb{E}_{x \sim \mathcal{X}}[\operatorname{Perf}(x; \pi_{\theta}, \mathcal{A})] - \Omega(\mathcal{A}) \quad \text{s.t.} \; \sum_{s \in \mathcal{I}} m(s) \leq \mathcal{C}_{\theta}, \; \mathcal{A} \cap \mathcal{I} = \varnothing$$

Jointly optimizes policy parameters and active skill set under a finite memory budget — makes explicit that internalizing a skill consumes parametric capacity $m(s)$, while keeping it external incurs routing/context cost $\Omega$.

Marginal External Contribution (Eq. 4):

$$\Delta_t(s) = \operatorname{Perf}(\mathcal{V}_t(s); \mathcal{A}_t) - \operatorname{Perf}(\mathcal{V}_t(s); \mathcal{A}_t \setminus \{s\})$$

Local counterfactual: the performance drop on validation tasks routed to $s$ when $s$ is withheld. Positive = externally valuable; zero/negative = absorbed or redundant.

Retention (Eq. 5): Keep $s$ if $\bar{\Delta}_t(s) \geq \tau_{\mathrm{keep}}$ — smoothed value exceeds external support cost threshold.

Retirement (Eq. 6): Remove $s$ if $\bar{\Delta}_t(s) < \tau_{\mathrm{retire}}$, exposure $u_t(s) \geq n_{\min}$, streak $\ell_t(s) \geq p$ — three safety conditions prevent premature removal of low-frequency skills.

Expansion (Eq. 7): Add new skill when persistent failures exceed $\tau_{\mathrm{expand}}$ with low current MEC — fills capability gaps rather than accumulating blindly.

6 minimum checks #

#	Check	Status
1	Assumption A.1 (retrieval recall) testable?	Yes — requires high-similarity skills to be in candidate set with prob ≥ $1-\delta_{\mathrm{ret}}$; empirically validated by retrieval hitting near-100% on routed validation
2	Assumption A.3 (GRPO local alignment) plausible?	Standard clipped-surrogate monotonic improvement under small step; standard in PPO theory
3	Assumption A.4 (MEC concentration) realistic?	Holds under bounded rewards + sufficient validation samples; invokes Hoeffding
4	Lemma A.8 (main lifecycle guarantee) conservative?	Yes — retain threshold set above $B_{\mathrm{op}} + \varepsilon_{\mathrm{val}}$, making false retirement improbable but potentially overly cautious
5	Lemma A.9 (patience protection) empirically verified?	Partially — Fig. 5 shows low-frequency skills (cle_003) surviving despite infrequent selection, consistent with exponential protection
6	Global convergence guarantee?	Absent — analysis provides local sufficient conditions only; no global optimality or convergence rate bound

The theoretical analysis is local and conservative: it shows that lifecycle operations are unlikely to worsen the objective under the stated assumptions, but does not guarantee convergence to an optimal active set. The assumptions (bounded rewards, concentrated estimates, sufficient retrieval recall) are reasonable but not formally verified on the benchmarks. This is common for agent papers.

§5 实验与数据 #

Main results #



Paper's Table 1, verbatim. Success rate comparison across prompt-based, memory-based, and RL-based methods on both benchmarks.

Key observations from Table 1: (1) SLIM† achieves 87.5% on ALFWorld, a 12.5-point gap over the next best RL method (SkillRL† 75.0); (2) On SearchQA, SLIM and SLIM† both reach 41.0%, indicating the benefit is internalized into the policy rather than requiring inference-time skill access; (3) naive skill insertion (†) sometimes hurts — GRPO† barely improves over GRPO, and Zero-Shot† degrades on SearchQA.

Ablation study #



Paper's Table 2, verbatim. Ablation removing each lifecycle component shows all three (retirement, expansion, contribution-aware auditing) are necessary.

The largest ablation drop (-18.7 points) comes from Random Audit, confirming that lifecycle decisions must be contribution-aware. Removing retirement alone drops 14.1 points (degenerates toward accumulation). Removing expansion loses 8.6 points (cannot fill capability gaps). Fixed Active Set Size (75.6) rules out a simple prompt-budget explanation.

Skill lifecycle case study #



Paper's Figure 5, verbatim (caption: "Case study of skill lifecycle on ALFWorld. Panel (a) plots selection count against marginal external contribution for retained, retired, and internalized skills. Panel (b) reports leave-one-skill-out validation bars. Panel (c) shows retired skill cases.").

Figure 5 reveals that frequency is not a proxy for external value: cle_003 is globally infrequent but locally indispensable (MEC 0.250), while pic_002 and gen_011 are frequent but nearly internalized (MEC 0.062, 0.080). This validates the MEC-based lifecycle over simpler frequency or recency heuristics.

Training dynamics #

• SkillRL: skills grow monotonically 38 → 73; final performance below SLIM
• Skill0: forced to 0 skills; validation collapses 92.2% → 76.6% at the zero-crossing
• SLIM: non-monotonic 38 → 46 → 21; no-skill policy improves to 84.4% while with-skill peaks 93.8%

Robustness #

Bootstrap 95% CIs (10,000 resamples): SLIM† – Skill0 gap = +13.3 [+3.9, +22.7] on ALFWorld; +1.72 [+1.13, +2.31] on SearchQA. Even from an empty skill bank, expansion alone reaches 76.4%.

§6 论证链 #

Step	Claim	Evidence	Dep
1	Monotonic skill paradigms (accumulation or elimination) are suboptimal under finite parametric capacity	Skill0 collapses 92.2→76.6% at zero-crossing; SkillRL accumulates 73 skills but scores only 75.0% (Fig. 3)	—
2	Leave-one-skill-out MEC provides an actionable local signal for external value	Ablation: Random Audit drops 18.7 points; Fixed Active Set drops 11.9 points vs SLIM (Table 2)	1
3	Contribution-aware retain/retire/expand yields non-monotonic trajectory and higher final performance	SLIM† achieves 87.5% with 21 active skills; simultaneous policy improvement to 84.4% no-skill (Table 1, Fig. 3)	2
4	External boundary is domain-dependent: some skills resist internalization	ALFWorld gap SLIM vs SLIM† = 14.8 points (procedural skills needed externally); SearchQA gap = 0 (fully absorbed)	3
5	Method is robust to initialization and generalizes to held-out tasks	Noisy bank → 85.6%; empty bank → 76.4% via expansion; held-out SearchQA sources improve +2.3 over Skill0 (Tables A1, A3)	3

§7 实现 cross-reference #

[实现未公开]

The paper does not release a public code repository. Implementation details are specified at the algorithm level (Algorithm 1 in §5) with the following key parameters:

• Backbone: Qwen3-4B (inference and training)
• Retrieval: Qwen3-Embedding-0.6B, $K=3$, $\tau_{\mathrm{emb}}=0.45$
• Lifecycle audit: every $d=10$ GRPO steps, top-$M$ skills by routed usage ($M=4$ ALFWorld, $M=12$ SearchQA)
• Thresholds: $\tau_{\mathrm{retire}}=0.001$, $\tau_{\mathrm{keep}}=0.03$–$0.05$, patience $p=3$, $n_{\min}=20$–$30$
• Expansion: o3 backbone with Anthropic-style skill-creator workflow, max 2–3 new skills per audit
• Training: GRPO without KL loss or KL-in-reward, LR $10^{-6}$, 120–180 steps

关键实现细节:

1. SLIM disables KL regularization entirely (both KL loss and KL-in-reward) — unusual for PPO/GRPO methods. This means lifecycle management alone constrains behavioral drift, since external skills act as an implicit anchor for covered behaviors.
2. Audit candidate selection uses routed usage (how often a skill was actually retrieved and used in rollouts), not skill-bank position or creation order. This ensures lifecycle decisions are made on skills with sufficient statistical evidence rather than arbitrarily sampling from the full bank.