SkillRouter: Skill Routing for LLM Agents at Scale — L2 #

§1 TL;DR #

In ~80K-skill agent pools, hiding skill body text drops routing accuracy 31–44pp. SkillRouter, a 1.2B full-text retrieve-and-rerank pipeline, reaches 74.0% Hit@1—beating a 16B base at 13× fewer params and 5.8× lower latency—via false-negative filtering and listwise reranking.

§2 痛点 · 方法 · 结果 #

Q1 痛点 #

Agent skill registries are scaling to tens of thousands of entries with heavy functional overlap. Deployed agent stacks (Claude Code, Codex, OpenClaw) practice progressive disclosure: the routing component sees full skill text, but only surfaces names and descriptions to the downstream agent. The critical question — whether metadata alone suffices for upstream routing at scale — had not been systematically tested. Prior tool-retrieval work (CRAFT, ToolRerank) operates on metadata in much smaller candidate pools (hundreds to low thousands), leaving the large-pool, high-overlap regime unexplored.

Q2 方法 #

SkillRouter is a two-stage full-text retrieve-and-rerank pipeline:

Stage 1 — Bi-encoder retrieval. Fine-tuned Qwen3-Emb-0.6B on 37,979 synthetic (query, skill) pairs. Each skill is encoded as name | description | body (body truncated to 2,500 chars). Hard negatives mined from 4 complementary sources: semantic neighbors (4), BM25 matches (3), taxonomy distractors (2), random (1). Trained with in-batch InfoNCE loss ($\tau = 0.05$). At inference, retrieves top-20 from the pre-embedded ~80K pool via ANN search.

Stage 2 — Cross-encoder reranking. Fine-tuned Qwen3-Rank-0.6B scores each query–candidate pair using flattened full skill text (body truncated to 2,000 chars, 4,096 max tokens). Trained with listwise cross-entropy on 32,283 candidate lists of 20 skills each.

Two critical training adaptations:

1. False-negative filtering — 3-layer filter prevents functionally equivalent skills from corrupting contrastive learning: (a) name deduplication, (b) body-text trigram Jaccard > 0.6, (c) embedding similarity > 0.92. Removes ~10% of mined negatives, contributing +4.0pp Hit@1.
1. Listwise reranking loss — When the retriever narrows to 20 topically plausible candidates, pointwise binary classification cannot discriminate. Listwise cross-entropy forces relative comparison among candidates. This single choice accounts for +30.7pp Hit@1 over pointwise BCE.

核心技术壁垒: Listwise reranking for homogeneous candidate pools. After retrieval, all top-20 candidates are topically plausible, making independent scoring collapse (pointwise: 43.3% Hit@1, worse than no reranker at 65.4%). The listwise formulation's requirement for inter-candidate comparison is the hardest-to-replicate insight — it requires both the right loss and carefully filtered training data where false negatives are removed.

Q3 结果 #

Metric	SkillRouter (1.2B)	Best base (16B)	Gap
Avg Hit@1	74.0%	68.0%	+6.0pp
Parameters	1.2B	16B	13× fewer
Median latency	496ms	2,900ms	5.8× faster
GPU memory	18,976 MB	22,539 MB	−15.8%
Downstream task success	27.56%	25.78%	+1.78pp

Gains generalize to SkillBench-Supp (256 queries, 77K pool, same checkpoints without retuning): 1.2B pipeline edges out 16B base on Hit@1 (.641 vs .637).

§3 架构 · 方法图 #



Paper Figure 2: SkillRouter pipeline. A bi-encoder retrieves top-20 candidates from the full ~80K pool; a cross-encoder reranks them. Both stages use full skill text (name + description + body).

The pipeline operates as an upstream retrieval component, not an agent loop. The downstream agent receives only skill names and descriptions from the top-ranked result — the full body is consumed only by the routing stages.

flowchart LR Q["User Task Query"] --> E["SR-Emb-0.6B\n(Bi-Encoder)"] S[("~80K Skill Pool\n(Pre-embedded)")] --> ANN["ANN Search"] E --> ANN ANN -->|"Top-20"| R["SR-Rank-0.6B\n(Cross-Encoder)"] R -->|"Ranked list"| INJ["Inject name+desc\ninto Agent Context"] INJ --> AGT["Downstream Agent\n(Claude / Kimi / glm)"]

Skill representation. Each skill is a structured 3-field document: name (median 3 tokens), description (median 21 words), and body (median 704 words, P90 = 1,991 words). The body constitutes 96.5% of skill tokens on average.

Training data flow. 37,979 synthetic queries generated by GPT-4o-mini from sampled skills (skill name excluded from prompt to prevent lexical leakage). 10 hard negatives per query from 4 sources, filtered through 3-layer false-negative removal. Reranker trains on 32,283 candidate lists produced by the trained encoder.

Agent-scope context. SkillRouter addresses single-shot upstream retrieval (query → ranked skill list), not multi-turn agent planning. It is fully automated with no human-in-the-loop. The downstream agents tested are general-purpose coding agents operating in open-ended task environments with tool access.

§4 作者证明 #

无形式化作者证明 — 仅实证。

All claims are supported by empirical evaluation rather than formal guarantees. The paper contains three loss definitions (standard IR formulations) and one attention computation formula, but no theoretical bounds on routing accuracy or convergence.

符号表 #

Symbol	Definition
$q$	User task query
$\mathcal{S} = \{s_1, \ldots, s_N\}$	Skill pool ($N \approx 80$K)
$\mathcal{G}_q \subseteq \mathcal{S}$	Ground-truth skill set for query $q$
$f(q, s)$	Cross-encoder reranker score
$\tau$	Temperature (0.05 encoder, 1.0 reranker)
$a_{i,\ell,f}$	Normalized attention share: query $i$, layer $\ell$, field $f$
$b_{i,f}$	Token-share baseline for field $f$

方程物理意义 #

Encoder loss (InfoNCE):

$$\mathcal{L}_{\text{enc}} = -\frac{1}{B}\sum_{i=1}^{B} \log \frac{\exp(\operatorname{sim}(q_i, s_i^+)/\tau)}{\sum_j \exp(\operatorname{sim}(q_i, s_j)/\tau)}$$

Contrastive loss pulling query–positive skill pairs together in embedding space while pushing apart in-batch negatives. Standard formulation; the novelty is in the training data (false-negative filtered hard negatives over full skill text).

Listwise reranking loss:

$$\mathcal{L}_{\text{LW}} = -\log \frac{\exp(f(q, s^+)/\tau)}{\sum_{j=1}^{K} \exp(f(q, s_j)/\tau)}$$

Softmax cross-entropy over the $K$-candidate list. The crucial difference from pointwise BCE: candidates compete directly against each other rather than being scored against an absolute threshold. This enables discrimination when all candidates are topically plausible.

Attention diagnostic:

$$a_{i,\ell,f} = \frac{\sum_{t \in T_{i,f}} \bar{A}_{i,\ell}(p_i, t)}{\sum_{f'} \sum_{t \in T_{i,f'}} \bar{A}_{i,\ell}(p_i, t)}$$

Normalized attention from the reranker's decision position to each skill field at each layer. Compared against token-share baseline $b_{i,f} = |T_{i,f}| / \sum_{f'} |T_{i,f'}|$ to test the length-only null hypothesis. The structured body → name → body trajectory (body 97.3% early → name 26.3% at layer 19 → body 98.1% final) argues against trivial length driving.

6 最低检查项 #

#	Check	Result
1	主要 claim 有对应实验支撑?	Yes — Tables 2–3 (routing accuracy), Table 5 (ablation), Table 6 (downstream transfer)
2	消融实验拆解关键组件?	Yes — false-negative filtering (+4.0pp) and listwise vs pointwise (+30.7pp) isolated in Table 5
3	基线公平比较?	Mostly — all use full skill text; 1.2B vs 16B asymmetric by design (the point is efficiency)
4	结果在多个 setting 复现?	Yes — Easy/Hard tiers, single/multi-skill, core + SkillBench-Supp, 4 downstream agents
5	局限性明确声明?	Yes — benchmark source diversity, FC@10 weakness, downstream scope bounded
6	数据泄漏检查?	Yes — benchmark skills excluded from training; SkillBench-Supp uses different LLM/prompt; 30 GT skills held out

Agent 特有检查 #

• 成功率矩阵: Hit@1 swept over encoder × reranker × scale × Easy/Hard × single/multi-skill. Monotonicity holds for scale (0.6B → 8B) but breaks for reranker addition (Qwen3-Emb-8B × Qwen3-Rank-0.6B drops below encoder-only).
• 延迟预算: 496ms median end-to-end (19.8ms encoder + ANN + ~476ms reranking of 20 candidates). No interactive-latency claim.
• 失败模式分类: Three modes: (a) recall failure at encoder stage (33/150 cases, dominant), (b) multi-hop prerequisite inference (invoice-fraud → PDF table extraction), (c) specialized multi-disciplinary vocabulary (HVAC case). Pipeline targets (a); modes (b) and (c) remain open.
• 可被 bounded 的量: Hit@1 could theoretically be bounded as a function of encoder R@$K$ and reranker accuracy conditional on correct recall, but no such analysis is provided.

§5 实验与数据 #

5.1 Metadata vs full text #



Paper Figure 1. Left: removing skill body text drops Hit@1 by 31–44pp across BM25, dense encoder, and pipeline. Right: layer-wise attention shows a body→name→body trajectory; name (3.0% tokens) peaks at 26.3% attention at layer 19, while the final layer returns to 98.1% body attention.

The body-access finding is the paper's motivational anchor. Three diagnostic controls argue it is not a simple artifact:

1. Attention redistribution across layers is structurally non-proportional to token length — the name field receives 8.8× its token share at layer 19.
2. Final-layer body attention is effectively uncorrelated with absolute body length ($r = 0.04$).
3. The nd → full gap remains ≥26pp even for skills with the longest, most detailed descriptions (Q4 quartile: >35 words, +31.8pp gap), ruling out a description-quality confound.

5.2 Main pipeline results #



Paper Table 3: End-to-end retrieve-and-rerank results (Easy/Hard averaged). SR-Emb-0.6B × SR-Rank-0.6B (1.2B total) reaches 74.0% avg Hit@1, vs 68.0% for the 16B base pipeline.

Fine-tuning compensates for a 13× parameter gap: SR-Emb-0.6B alone (65.4% Hit@1) edges past Qwen3-Emb-8B (64.0%). Adding the tuned reranker further boosts to 74.0%. The improvement is consistent across both Easy (+8.0pp) and Hard (+4.0pp) tiers, though the Hard-tier gap narrows due to targeted distractors.

Reranker contribution decomposed (150 Easy+Hard evaluations): fixes 19 cases (12.7%) while degrading only 6 (4.0%) → net +8.7pp Hit@1 gain. The 33 remaining misses are recall failures or multi-hop reasoning requirements.

5.3 Two essential training choices #



Paper Table 5: Ablation isolating false-negative filtering (+4.0pp encoder Hit@1) and listwise vs pointwise reranking (+30.7pp pipeline Hit@1).

The pointwise collapse is the most striking result: fine-tuning with binary cross-entropy actively degrades to 43.3% Hit@1 — 22pp below the no-reranking encoder-only baseline (65.4%). In the homogeneous top-20 pool, pointwise scoring assigns similar probabilities (~0.52) to all topically related candidates without relative discrimination.

False-negative filtering has a smaller but critical effect: without it, 39,065 functionally equivalent skill pairs (~10% of mined negatives) poison the contrastive signal, costing 4.0pp encoder Hit@1 and 3.1pp MRR@10.

5.4 Downstream agent evaluation #



Paper Table 6: Task success across 4 coding agents (3 trials averaged), 75-task core set. SkillRouter recovers 71–73% of the no-skill → gold-skill uplift vs 59–61% for the base pipeline.

Routing improvements transfer to downstream execution: +1.78pp (top-1) and +2.33pp (top-10) over the 16B base. The benefit is more pronounced for stronger agents — Claude Sonnet/Opus 4.6 gain +3.22pp while glm-5 and Kimi-K2.5 gain only +0.89pp, consistent with a ceiling on routing utility for agents that cannot fully exploit correctly routed skills.

Both top-1 and top-10 retrieval yield essentially identical overall success (~27.6–27.8%), suggesting diminishing returns from expanding the shortlist beyond a quality threshold.

5.5 Where SkillRouter loses #

Three settings where the 16B base pipeline outperforms:

• Multi-skill FC@10: Qwen3-Emb-8B × Qwen3-Rank-8B (.382) > SR pipeline (.353). The listwise loss optimizes for top-1 discrimination at some cost to exhaustive set recovery.
• Specialized multi-disciplinary domains (HVAC case): Base pipeline recovers all 5 gold skills in top-10 while SR recovers only 1. Precise vocabulary matching across multiple sub-disciplines favors the larger model's broader knowledge.
• glm-5 downstream top-1: Base (24.89%) > SR (23.11%). The weaker agent cannot exploit the routing advantage.

5.6 Benchmark and evaluation details #

• Primary benchmark: 75 expert-verified queries, ~80K pool (78,361 Easy + 780 Hard distractors). 24 single-skill, 51 multi-skill. 55 domains across 8 super-categories.
• SkillBench-Supp: 256 queries (168 descriptive + 88 indirect), 77K pool, 100 GT skills from 3 sources (OpenClaw, AgentSkillOS, pool-sampled). Indirect queries consistently harder across all systems.
• Evaluation protocol: Hit@1 (primary), MRR@10, R@$K$ ($K \in \{10, 20, 50\}$), FC@10. Multi-skill Hit@1 = any GT at rank 1.
• Baselines tested: 11 encoder models (BM25, 3 traditional, 3 decoder, 2 proprietary, 2 SR variants) × 7 reranker configs + 2 LLM-as-judge.

§6 论证链 #

Step	Claim	Evidence	Direction
1	Metadata-only routing fails at scale	31–44pp Hit@1 drop across 3 method families; BM25-nd = 0.0% (§3, Fig 1)	§3 → §4 motivation
2	Failure is not a length/description-quality artifact	Name 26.3% attention at layer 19 despite 3% tokens; $r = 0.04$ with body length; gap ≥26pp for longest descriptions (§3, App C/D)	§3 controls → §4 design
3	Compact full-text pipeline beats larger base	1.2B: 74.0% vs 16B: 68.0% Hit@1 at 5.8× lower latency (§5.2, Tab 3)	§4 design → §5.2
4	False-negative filtering prevents training data corruption	Clean vs raw negatives: +4.0pp Hit@1, +3.1pp MRR@10 (§5.3, Tab 5)	§4 adaptation → §5.3 ablation
5	Listwise loss essential for homogeneous pools	Listwise 74.0% vs pointwise 43.3%; pointwise worse than no reranker (§5.3, Tab 5)	§4 adaptation → §5.3 ablation
6	Gains transfer to agent execution	+1.78pp task success across 4 agents; stronger agents benefit more (§5.5, Tab 6)	§5.2 routing → §5.5 downstream
7	Results generalize to independent benchmark	SkillBench-Supp: 1.2B edges 16B without retuning (.641 vs .637, §5.4)	§5.2 → §5.4 external validation

Weakest links: Step 6 shows a modest +1.78pp absolute gain, and the gold-skill ceiling is only 32.67%, limiting the resolution of the downstream evaluation. Step 7's supplementary benchmark is single-label only. The paper acknowledges FC@10 disadvantage (Step 3 qualifier): the compact model's top-1 strength does not extend to exhaustive multi-skill recovery.

§7 实现 cross-reference #

[实现未公开] — No code repository released. Base models: Qwen3-Emb-0.6B/8B, Qwen3-Rank-0.6B/8B. Hardware: single NVIDIA GPU, 96GB HBM3 (Hopper-generation). Both stages train for 1 epoch.

关键实现细节 #

1. False-negative filtering thresholds. 3-layer filter: (a) exact name match → 24,879 pairs, (b) body trigram Jaccard > 0.6 → 13,860 pairs, (c) base encoder cosine > 0.92 → 326 pairs. Total 39,065 pairs (~10% of negatives). These thresholds are not individually ablated; Jaccard and cosine cutoffs are potential sensitivity points for replication.
1. Asymmetric truncation budget. Encoder: query 1,500 chars, description 300 chars, body 2,500 chars, max 2,048 tokens. Reranker: description 500 chars, body 2,000 chars, max 4,096 tokens. The reranker sees a tighter body but has more total token budget per candidate. This asymmetry matters because the encoder processes each skill independently while the reranker must fit 20 candidate comparisons sequentially.

核心技术壁垒 补充 #

The listwise reranking effectiveness depends on the upstream encoder producing a top-20 list where the correct skill is present (R@20 = 75.4%) but not at rank 1 — creating reranking headroom. If encoder R@20 drops (NV-Embed-v2: 56.2%), the reranker has nothing to rescue. The pipeline's success is load-bearing on both stages: the encoder must achieve high recall with clean contrastive training, and the reranker must learn relative discrimination under listwise competition. Replicating either stage alone fails — pointwise-only collapses, and encoder-only leaves +8.7pp on the table.