IntentKV: Cross-Turn Intent-Aware KV Cache Pruning for Agent Inference — L2 #

1. TL;DR #

Multi-turn agents blow up KV memory/bandwidth far more than compute. IntentKV keeps the base LLM frozen and prunes KV by (a) scoring history against a session-level QueryMemory of accumulated intent + a zero-init residual head, and (b) evicting via slot-map redirection to a dead slot instead of compaction — so pruning stays compatible with radix prefix reuse.

2. Q1 / Q2 / Q3 #

Q1 — 痛点 (why). In browsing / deep-research / tool-use agents, a short user query fans into a long trajectory of retrieved docs, tool outputs, and intermediate reasoning. KV memory capacity and KV read bandwidth — not parameter compute — become the serving bottleneck. Existing single-prompt pruners (StreamingLLM, SnapKV, H2O) break two agent assumptions: (i) token importance drifts across turns, so a ranking from one prompt goes stale and discards future-use evidence; (ii) standard compaction relocates surviving KV rows, destroying the prefix identity that radix/prefix caches (SGLang, vLLM) reuse across turns. Agents are thus forced to choose either pruning or prefix reuse.

Q2 — 方法 (how). IntentKV separates what to keep from how the pruned state is laid out:

• Retention: a per-session QueryMemory $\mathbf{M}_t$ accumulates query-like signals across turns via a geometric (EMA) recurrence and scores each history token by memory-attention. A small zero-initialized residual head (215k params) adds a learned correction on top, grounded on which positions get re-attended by future tool calls.
• Layout: eviction is a slot-map redirection — dropped logical positions point to a single sentinel "dead slot" (K set to $-10^4$, V set to $0$), while surviving K/V rows, RoPE phases, and slot identities stay put. Numerically a hard mask, but at slot-map level so flashinfer/FA3 kernels run unmodified and the radix tree still matches the prefix.

核心技术壁垒 (the single hardest-to-replicate insight): making per-request KV pruning composable with radix-prefix reuse by never relocating survivors. Both the paged slot map and the sentinel-mask idiom are borrowed; the non-obvious, hard-won contribution is the alias-aware deallocation discipline (§7) that lets a dropped slot be freed only when it is not the sentinel, not radix-protected, and not aliased by a sibling request or a prior redirect — without this exact condition, freeing slots corrupts sibling KV reads, which is precisely why prior compaction pruners had to disable prefix caching.

Q3 — 结果 (so what). On BrowseComp-Plus (830 queries) at an 8k budget, IntentKV-Phase-2 lands within 0.96 True-Acc points of the no-pruning ceiling on Qwen3-8B and beats the strongest heuristic by 10.36 points on Qwen2.5-14B, while holding a 20.7% prefix-hit rate where compaction baselines collapse to 0–3%. On the 100 longest queries it cuts worst-case peak request tokens by 77.8–81.7% and worst-case raw KV reads by up to 92.6%.

3. 架构 / 方法图 #

Paper's Figure 2 (caption: "IntentKV method overview"). The figure traces one compression event: ① the agent context (system prefix + history + current query span) feeds ② a QueryMemory update, then ③ scoring combines (a) the rule score, (b) the residual correction, and (c) their sum; low-scoring positions are redirected to the dead slot while kept tokens and their slots are untouched. This is the load-bearing diagram — it shows the retention/layout factorization that is the paper's whole thesis.

Motivation figure — why cross-turn matters. The stale-ranking failure mode:

Paper's Figure 1 (caption: "Prompt-local KV pruning becomes stale in multi-turn agents"). Panel (a) shows the fan-out from single-shot to multi-turn; panel (b) contrasts signal-mismatch (a one-shot scorer drops "future-use evidence") with IntentKV's session-aware retention. The reader should notice that the oldest tokens (initial request, first retrieval) are exactly the ones a recency/time-decay prior would evict but agents later need.

The agent loop and where IntentKV intervenes #

IntentKV is not an agent policy — it intervenes once per request, after prefill, when sequence length $N$ exceeds budget $C$. The surrounding agent loop is a standard ReAct think-act-observe cycle with the KV compressor bolted into the serving layer:

• Memory model: short-term = the (pruned) context window; the session-level QueryMemory $\mathbf{M}_t$ is a compact per-session intent state (one query-row shape), stored in an LRU of 1,024 memories per compressor. No vector DB / episodic log — this is intra-cache memory that composes with orchestration-layer memories (MemGPT / A-MEM) rather than replacing them.
• Tool invocation: native OpenAI Chat-Completions function calling with JSON arguments (two tools: snippet retrieval + full-document fetch); structured JSON is robust to mid-string truncation.
• Error recovery: the rule scorer is a strict floor under the learned head (§4), so if the residual is unhelpful the system degrades to the heuristic rather than failing; on single-turn inputs IntentKV falls back to its heuristic scorer.

Planning & reasoning #

• Planning style: predetermined ReAct loop (think → act → observe); IntentKV adds no new planning — it only changes the KV state the loop runs against.
• Budget: at most 32 tool-use turns per query; deterministic decoding ($T{=}0$). The KV budget $C \in \{8192, 16384\}$ caps compressible history, not total prompt+generation tokens.
• Backtracking: none at the agent level; at the cache level eviction is one-way per compression event (a dropped position is redirected to the dead slot and cannot be revived).

4. 作者证明 #

无形式化作者证明 — 仅实证. IntentKV has no convergence or success-rate guarantee. The one formal claim is an initialization/lower-bound argument for the residual head, plus a numerical-masking equivalence for the sentinel. What could have been bounded but is not: worst-case accuracy loss vs. budget $C$, or a retention-recall guarantee against future-attended positions.

Notation table.

方程物理意义 (load-bearing equations).

• Top-$k$ selection: $\mathcal{K}^\star=\mathcal{F}\cup\operatorname*{arg\,top}_{|\mathcal{K}|=C^\star}\sum_{j\in\mathcal{K}} s_j$ — greedily fill the residual budget with highest-scoring history, always keeping the forced set.
• Memory recurrence: $\mathbf{M}_t = e^{-\lambda}\mathbf{M}_{t-1} + \mathrm{Enc}(\mathbf{q}_t)$, then renormalized to unit norm along the head dim — accumulates evolving intent; without renorm the norm grows ~linearly with turns and the score softmax collapses to one coordinate.
• Rule score: $\mathrm{rule}_j=\sum_{l,h} a_{l,h,j}$ with $a_{l,h,j}=\mathrm{softmax}_{\mathcal{H}}(\mathbf{M}_t[l,h]\cdot\mathbf{K}[l,h,j]/\sqrt{D})$ — scores each token by how strongly accumulated intent attends to it; replaces SnapKV's $W$ tail-Q rows with one state, cutting cost from $\mathcal{O}(WLH_qN)$ to $\mathcal{O}(LH_qN)$.
• Residual: $s_j=\mathrm{rule}_j+\alpha\cdot\mathrm{MLP}(\bm{\phi}_j\oplus\mathbf{c}_j)$ — rule is primary; residual is an additive correction.
• Redirect: $\mathbf{S}_r[j]\leftarrow s^\dagger$ for all dropped $j$ — eviction without compaction.

6 minimum checks.

1. Dimensional/shape check: $\mathbf{M}_t$ has the shape of one post-RoPE query row ($L\times H_q\times D$); scoring against $\mathbf{K}[l,h,j]$ requires GQA KV-heads expanded to $H_q$ — stated explicitly. ✔ consistent.
2. Initialization floor: with $(\alpha{=}1,\ \mathrm{fc}_2{=}0)$ the residual output is 0, so $s_j=\mathrm{rule}_j$ exactly on day zero; gradients still flow through $\mathrm{fc}_2$ (whereas $\alpha{=}0,\mathrm{fc}_2{=}0$ is a dead $(0,0)$ saddle). ✔ the argument is internally sound — this is why "rule scorer is a strict floor."
3. Numerical masking equivalence: sentinel pre-softmax logit $=-10^4\sqrt{D}$, so softmax weight $=\exp(-10^4\sqrt{D})$, far below smallest bf16/fp16 value; $V{=}0$ nulls any residual contribution. ✔ numerically a hard mask.
4. Budget semantics: $C$ caps compressible history $\mathcal{H}$, not total tokens; forced set is always retained. Verified against the 14B result — at 16k budget, ~17k-token trajectories "rarely trigger eviction," matching the observed narrowing of IntentKV's advantage. ✔ self-consistent.
5. Parameter count: $\bm{\phi}_j\in\mathbb{R}^{3D+1}$, MLP hidden 256, cross-attn $H_c{=}4$/$d_c{=}128$; for $D{=}128$ the head is 214,274 params. ✔ plausible for a 2-layer GELU + small cross-attn.
6. Alias-safety: a dropped slot $u$ is freed only if $j$ is outside the radix-protected prefix, $u\neq s^\dagger$, and $u\notin\{\mathbf{S}_r[i]:i\in\mathcal{K}^\star\}$ — condition (c) catches prior-redirect aliases and sibling-shared slots. ✔ this is the correctness lynchpin for combining pruning + prefix reuse.

Agent-specific asks.

• Success-rate model (empirical sweep): the effective sweep axes are (backbone × budget × scorer variant). Monotonicity holds along budget (16k ≥ 8k True Acc for IntentKV) and along scorer richness at 16k (memory > query-only). Notably non-monotone at 8k: the learned residual hurts by 0.60 points because the rule prior already "saturates the dead-slot ceiling" (see §5 ablation).
• Latency budget per turn: dominated by prefill + decode over the (pruned) KV; the compression event itself is $\mathcal{O}(LH_qN)$ scoring + a tensor-parallel all-reduce of the score vector. The paper reports wall-clock (Table 2) rather than per-turn latency; it does not claim "interactive latency" but shows up to 43% wall-time reduction vs. compaction baselines.
• Failure-mode taxonomy: the paper's own retention has no explicit failure taxonomy, but the cross-architecture study identifies four failure classes (F1 refusal-to-use-tools, F2 single-shot tool bias, F3 hardware/quant mismatch, F4 parser incompatibility); the dominant one is F1 (models never emit a tool call), and it is not a KV-compression failure — it sits at the agentic-behavior layer.

5. 实验与数据 #

Headline accuracy (BCP, 830 queries).



Paper's Table 1. At 8k, compaction baselines collapse on completion (StreamingLLM 65.4%, SnapKV/H2O ~45.7%), dragging True Acc down, while IntentKV holds 84.6% completion → 14.10 True Acc on Qwen3-8B and 18.55 vs. StreamingLLM's 8.19 on the 14B. The key reader takeaway: the win is a completion win — baselines run out of KV / hit turn caps before answering.

Efficiency and prefix reuse.



Paper's Table 2. IntentKV's Raw KV Reads match Full-cache (32.0M vs 32.2M on Qwen3-8B/16k) and are ~2× lower than every compaction baseline on the 14B, with wall time up to 43% lower than H2O. Notice IntentKV is often cheaper than baselines at equal or better accuracy — the tighter live working set plus preserved prefix reuse compound.

Pareto frontier under two cost metrics.



Paper's Figure 4. Eight panels: True Acc vs. peak request tokens (top row) and vs. effective live KV (bottom row). IntentKV (★) sits up-and-left of every heuristic in the 8k panels, i.e. higher accuracy at lower memory — the frontier the headline single-number metric obscures.

Worst-case stress test.



Paper's Table 3. On the heaviest 100 queries, IntentKV-8k cuts worst-case peak request tokens from 92.3k→20.5k and raw KV reads from 411M→31M (↓92.6%) on the 14B, while True Acc stays within ~2 points. This is the strongest argument that IntentKV absorbs tail trajectories at a 1–2 order-of-magnitude smaller footprint.

Layout ablation (isolating the substrate).



Paper's Table 7. Re-running SnapKV/H2O on IntentKV's dead-slot substrate (same scorer, same budget) cuts their wall-time 44–46% and Raw KV Reads 39–47%, and raises their True Acc 2.4–5.5 points — proving the layout gain is separable from the QueryMemory scoring gain. Under the matched substrate IntentKV still retains a 1.5× Raw-KV-read reduction over SnapKV at statistically indistinguishable accuracy ($\Delta{=}0.35\sigma$).

Evaluation details. Benchmark: BrowseComp-Plus (830-query deep-research over ~100K docs) + cross-benchmark FRAMES (824 multi-hop QA, adapted into a multi-turn stress profile). Metric: True Acc = Correct/Total (non-completion counted wrong), judged by a Qwen3-32B LLM. Baselines: Full-cache ceiling, StreamingLLM, SnapKV, H2O, TrimKV.

6. 论证链 #

Step	Claim	Evidence (paper-internal)
1	Agent trajectories make KV memory/bandwidth the bottleneck, not compute.	§1 asymmetric profile; worst-case 92–115k peak tokens in Table 3.
2	Single-prompt pruners fail on agents for two reasons: stale rankings and lost prefix identity.	Figure 1 dual failure mode; Figure 3 attention drift across $q_0\ldots q_4$.
3	Therefore separate retention (what to keep) from layout (how to store), so each can be fixed independently.	§3 factorization; Table 7 shows the two axes are independently attributable.
4	Cross-turn QueryMemory + zero-init residual retains future-use tokens better than one-shot scorers.	Table 1 completion/True-Acc wins; §5 ablation: removing memory costs up to 3.13 True-Acc points.
5	Slot-map redirection preserves prefix identity, so pruning composes with radix reuse.	20.7% prefix-hit at 8k vs. 0–3% for compaction; Table 2 matched Raw KV Reads vs. Full-cache.
6	Net effect: near-ceiling accuracy at 1–2 orders-of-magnitude smaller worst-case footprint.	Table 3: ↓77.8–81.7% peak tokens, ↓ up to 92.6% raw KV reads, True Acc within ~2 points.

7. 实现 cross-reference #

核心技术壁垒 detail — alias-aware deallocation. The subtlety that makes pruning composable with prefix reuse: a dropped physical slot $u=\mathbf{S}_r[j]$ is returned to the allocator only if (a) $j$ is outside the radix-protected prefix, (b) $u\neq s^\dagger$, and (c) $u\notin\{\mathbf{S}_r[i]:i\in\mathcal{K}^\star\}$. Condition (c) is the hard part — it catches both positions already redirected to the sentinel in a prior compression round and slots shared with sibling requests on the same radix branch. Getting this wrong silently corrupts sibling KV reads, which is exactly why earlier compaction pruners disabled prefix caching. The slot indirection is from PagedAttention and the sentinel-as-mask is a standard idiom; the contribution is the integration discipline, not either primitive alone.

关键实现细节 (easy-to-miss tricks).

1. QueryMemory unit-norm projection. After each EMA update $\mathbf{M}_t$ must be renormalized to unit norm along the head dim. Skip it and the memory norm grows ~linearly with turn count, collapsing the rule-score softmax onto a single coordinate — a silent failure that only manifests deep into long sessions. It is also softly enforced during training via the $\eta(\lVert\mathbf{M}_t\rVert_2-1)^2$ loss term.
2. Zero-init residual with $\alpha{=}1$ (not $\alpha{=}0$). Initializing the MLP output at 0 and $\alpha{=}1$ recovers the rule score exactly on day zero while keeping gradients flowing; the tempting $(\alpha{=}0,\mathrm{fc}_2{=}0)$ pairing freezes training at a saddle. Also: future-action labels come from literal substring matching against the next 5 tool-call arguments — rows with no matched evidence are dropped, not back-filled with a recency proxy, trading recall for label precision.

Session-id derivation (multi-tenant safety). Session key resolved in priority: explicit session_id → token tuple of a designated session span → hash of first $P{=}256$ input tokens → per-request key that never matches. Concurrent requests update disjoint keys, so multi-tenant serving never cross-contaminates intent (LRU of 1,024 memories per compressor).

Serving / sandboxing. All runs on SGLang with radix prefix caching; compaction baselines run with radix reuse disabled after eviction (to avoid wrong RoPE positions). IntentKV uses the same flashinfer/FA3 attention kernels — only the slot map differs; the dead-slot sentinel currently requires fp16/bf16 KV pools.

[实现未公开 — 部分] Code and the 214,274-parameter residual-head checkpoints are stated to be released under MIT (no base-LLM weights, ToolBench, or BCP content redistributed), but the L1 source carries no file:line citations to a public repository, so concrete call sites are not yet available.