SmoothAgent — L2 Distillation #

1. TL;DR #

Context engineering (offload/summarize/truncate/isolate) keeps agent contexts short but each rewrite invalidates the KV cache and forces a re-prefill, spiking TTFT. SmoothAgent observes these rewrites are segment-decomposable — each prefix segment transforms independently of future tokens — so it precomputes transformed KV caches asynchronously in a lookahead stream and splices them in at commit time. A best-effort SLO-aware scheduler admits lookahead work only within LC slack. Up to 11.9× lower transform-point TTFT.

2. Q1 / Q2 / Q3 #

Q1 — 痛点 (the problem) #

Long-horizon LLM agents accumulate context every turn (tool outputs, observations, history), which both raises cost/GPU-memory and degrades reliability through context rot — quality drops as context grows even inside the supported window. To fight this, production frameworks apply context engineering (reduction, offloading, isolation) to control length. But from a systems view every such technique is a context transformation: it rewrites the prefix, which invalidates the existing KV cache and forces a blocking re-prefill. The added latency — named context transformation overhead — lands directly on the critical path and produces sharp tail-TTFT spikes at each trigger, violating SLOs in latency-sensitive settings.

Agent scope: task class is open-ended coding/analysis (multi-step code reading via shell tools); interaction pattern is continuous multi-turn long-lived sessions; autonomy is fully autonomous per turn (no human-in-the-loop in the evaluated loop). This is an agent-serving / infrastructure paper, not a new agent reasoning method.

Q2 — 方法 (the method) #

Two coupled mechanisms plus a formal property:

1. Segment-decomposability (formal observation): a transformation $T$ over context $C$ partitioned into segments $\{S_1,\dots,S_n\}$ satisfies $T(C)=T(S_1)\,\|\,\cdots\,\|\,T(S_n)$ — each segment's transform depends only on itself (and prior state), never on future tokens. This is what licenses ahead-of-time execution.
2. Lookahead programming model: maintain a lookahead context alongside the working context. A 4-method strategy interface (should_lookahead / transform / should_commit / should_promote) lets the runtime asynchronously precompute transformed KV caches in a background stream, then at commit directly replace the prefix instead of re-prefilling. Developers keep their agent loop; they only implement the interface and insert two hooks.
3. Lookahead-aware scheduler: treats lookahead requests as best-effort (BE) jobs that never delay latency-critical (LC) requests, gated by a context-aware batch-latency model (Eq. 1) that — unlike token-budget-only models — accounts for KV-cache length in attention cost.

核心技术壁垒 (THE hardest-to-replicate insight): the isolation strategy reframes auto-regressive instruction decoding as a stream of fine-grained segments, so the sub-agent's KV cache is built incrementally during delegation-instruction generation. Because decode is slower than prefill, the sub-agent prefill is fully hidden behind the decode it overlaps — turning a bottleneck (slow decode) into free preparation time. Getting this right (segment granularity, prefix-dependency ordering, near-zero final commit prefill) is the subtlest engineering in the paper.

Q3 — 结果 (the results) #

• 11.9× transform-point TTFT improvement (summarization, the most expensive transform).
• Avg tail-TTFT reduction 62.0% (Qwen3-8B) / 61.5% (Qwen3-32B) in PD co-located.
• 64.5% avg reduction in PD disaggregated (up to 64 concurrent agents).
• 26.8%–80.5% transform-point TTFT reduction across 4 real frameworks (LangChain, LlamaIndex, AutoGen, OpenClaw).
• Performance model max relative error 13.7% vs 41.5% for a token-budget-only model.

---

3. 架构 / 方法图 #

The core execution timeline #



Paper's Figure 4 (caption: "Execution timeline of the lookahead programming model. The main stream proceeds normally while the lookahead stream asynchronously computes the transformed segments' KV cache in advance. At commit time, the pre-computed result is spliced."). The main stream runs the agent loop uninterrupted while the lookahead stream materializes transformed-segment KV caches off the critical path. The key thing to notice: the commit is a splice, not a prefill — the expensive work has already happened, so no TTFT spike appears at the trigger.

The overhead this eliminates #



Paper's Figure 3 (caption: "An illustrative example of context transformation overhead, including KV cache invalidation and additional transformation cost, which leads to TTFT spikes."). When earlier turns are summarized, the prefix changes, cached KV entries for that prefix become invalid, and the next response must re-prefill — this is the spike SmoothAgent hides.

The agent loop (agent-specific: state of one turn) #

Per the agent template, the one-turn loop is a state machine. SmoothAgent overlays a lookahead sub-loop on the standard observe→act loop:

stateDiagram-v2 [*] --> Commit Commit: on_commit (splice if ready, else promote+await) Generate: llm.generate(messages) ToolCall: execute_tool -> obs SegBoundary: on_segment_boundary (enqueue transform) Commit --> Generate Generate --> ToolCall: has_tool_call Generate --> SegBoundary: no_tool_call ToolCall --> SegBoundary SegBoundary --> Commit: not task_done SegBoundary --> [*]: task_done state LookaheadStream { Idle --> Transforming: should_lookahead() Transforming --> Idle: segment done Transforming --> Promoted: should_promote() Promoted --> Idle: committed }

Memory model: short-term = working context window (MainState.messages); long-term = file system (offloading writes observations to /path/to/fileN and leaves pointers); episodic = the growing multi-turn trace itself. Tool invocation: standard tool-call → execute_tool → observation appended (shell commands head/tail/sed in the eval). Error recovery: if a lookahead transform has not finished at the commit point, the runtime falls back to synchronous execution (promote priority, await) — i.e., worst case degrades gracefully to the baseline, never worse.

Planning & reasoning (agent-specific) #

Planning style is a predetermined workflow, not tree search or ReAct-style deliberation — the "intelligence" here is systems scheduling, not agent reasoning. Decomposition is top-down for isolation (main agent → sub-task instruction → sub-agent). Budget is context-length-driven: transforms trigger at token thresholds (e.g. offloading at 15K, summarization soft/hard at 11K/15K). Backtracking: none at the reasoning level; the only "undo" is that an unready lookahead result is simply discarded and recomputed synchronously.

Strategy instantiations #



Paper's Figure 5 (caption: "Offloading transforms bulky observations (e.g., tool outputs) into compact references to external storage. Since each completed observation can be rewritten independently, the transformed KV cache can be prepared ahead of time."). Offloading is lossless (exact fidelity, restorable) but yields the smallest speedup because after the first trigger the context is already compact.



Paper's Figure 8 (caption: "Context isolation launches a sub-agent with a clean context. By decomposing instruction generation into segments, the sub-agent context can be prepared ahead of time."). Sub-figure (b) shows the delegation instruction split into decode-time chunks (S2, S3); each chunk incrementally extends the sub-agent KV cache so almost all sub-agent prefill overlaps the (slower) instruction decode — the core-barrier trick from §2.

---

4. 作者证明 #

无形式化作者证明 — 仅实证 for the end-to-end speedup (no convergence/success guarantee — expected for an agent-serving paper; what could have been bounded is the commit-miss probability as a function of the soft-vs-hard threshold gap). However, the paper does carry two load-bearing analytical models that admit checks.

Notation table #

Symbol	Meaning
$C$	current accumulated context
$T(\cdot)$	context transformation function
$S_i$	$i$-th context segment; $\	$ = concatenation
$M$	total forward tokens in a batch
$B_{\mathrm{decode}}, B_{\mathrm{prefill}}$	decode / prefill request sets
$q_j$	query tokens in prefill chunk $j$; $\mathrm{prefix}_j$ its cached prefix len
$L_j$	KV-cache length of decode request $j$
$\alpha_{\mathrm{d}}, \alpha_{\mathrm{p}}$	profiled decode / prefill attention coefficients
$A_j$	per-chunk prefill attention work
$t_{\mathit{budget}}, \delta$	LC slack budget / TBT SLO bound

方程物理意义 #

Segment-decomposability: $$T(C)=T(S_1)\;\|\;T(S_2)\;\|\;\cdots\;\|\;T(S_n)$$ the whole-context transform equals concatenation of independently-transformed segments — independence from future tokens is exactly the license for lookahead.

Batch-latency model (Eq. 1): $$\mathrm{EstBatchLatency}(B)=T_{\mathrm{GEMM}}(M)+\alpha_{\mathrm{d}}\sum_{j\in B_{\mathrm{decode}}}L_{j}+\alpha_{\mathrm{p}}\sum_{j\in B_{\mathrm{prefill}}}A_{j}$$ decomposes cost into token-driven GEMM (lookup table), KV-length-driven decode attention (linear in $\sum L_j$), and query×prefix-driven prefill attention $A_j=q_j\cdot\mathrm{prefix}_j+\tfrac{q_j(q_j+1)}{2}$. This is the scheduler's admission oracle.

6 minimum checks #

1. Dimensional consistency: all three RHS terms of Eq. 1 are latencies (coefficients $\alpha$ carry time/token-work units); sum is a latency. ✓
2. Limiting case (no cached context): with $\mathrm{prefix}_j=0$ and $L_j\to 0$, attention terms shrink and Eq. 1 collapses toward pure $T_{\mathrm{GEMM}}(M)$ — recovering the token-budget-only model as a degenerate case. ✓ (explains why token-budget-only works for stateless but fails for long-KV agent workloads.)
3. Causal-mask arithmetic: $A_j=\sum_{i=0}^{q_j-1}(\mathrm{prefix}_j+1+i)=q_j\cdot\mathrm{prefix}_j+\tfrac{q_j(q_j+1)}{2}$ — the closed form matches the arithmetic-series sum. ✓
4. Monotonicity: $\mathrm{EstBatchLatency}$ increases in $M$, $L_j$, $\mathrm{prefix}_j$, $q_j$ — every added request/token can only raise estimated latency, so greedy admission (Alg. 1/2) is safe (admitting a chunk never decreases predicted cost). ✓
5. Slack safety: Alg. 1 admits BE chunk $c$ only if $\mathrm{EstBatchLatency}(B_{LC}\cup B_{BE}\cup\{c\})\le t_{budget}=\max(0,s_{min})$; since $t_{budget}$ is the min LC slack, no LC deadline is violated if the model is accurate. ✓ (conditional on check 6.)
6. Empirical model accuracy: Fig 15 — context-aware model max relative error 13.7% vs 41.5% token-budget-only, validating that check-5's guarantee rests on a sufficiently accurate oracle. ✓ (bounds the residual SLO risk.)

Agent-specific asks #

• Success-rate model: N/A — the paper optimizes latency, not task success; it holds accuracy fixed by replicating each framework's native strategy verbatim (only removing the blocking).
• Latency budget per turn: yes — the whole point. Per turn = LLM inference + tool call + (hidden) transform; the sweep is concurrency (1/4/8/16, up to 64) × strategy × model scale, and TTFT stays smooth (Fig 12) rather than claiming an absolute "interactive" bound.
• Failure mode classification: the dominant "failure" is a commit miss (lookahead not ready in time) → falls back to synchronous, mitigated by the soft-threshold and should_promote early-priority signal. No taxonomy of >1 class.

---

5. 实验与数据 #

End-to-end transform-point TTFT #



Paper's Figure 10 (caption: "Transform-point TTFT for each strategy on Qwen3-8B (PD co-located) at concurrency levels 1, 4, 8, and 16. SmoothAgent eliminates the transformation-induced TTFT spike across all strategies and concurrency levels."). The spike SmoothAgent removes is largest for summarization and smallest for offloading; note offloading's benefit shrinks as concurrency rises (bars converge), because repeated offloads operate on already-compact context.



Paper's Figure 12 (caption: "Context length (top) and per-turn TTFT (bottom) for a single Qwen3-8B agent under each strategy..."). This is the load-bearing case study: the top row shows each strategy capping context growth at its trigger, while the bottom row shows the baseline's TTFT spike at exactly that trigger — and SmoothAgent's flat profile because the cost was amortized across preceding turns. Triggers: offloading/summarization at 15K, keep-recent-K at turn 22, summarization soft threshold at 11K.

PD disaggregated scale-out #



Paper's Figure 13 (caption: "Transform-time TTFT on Qwen3-8B in a PD disaggregated deployment with four prefill and four decode instances."). Absolute TTFT is higher than co-located (KV transfer + routing overhead) but the relative spike elimination holds (avg 64.5%), showing lookahead survives the disaggregated setting where NIXL moves KV between instances.

Scheduler isolation & model validation #



Paper's Figure 14 (caption: "Impact of lookahead traffic on latency-critical requests. Our lookahead-aware scheduler effectively reduces the interference as lookahead traffic increases."). Vanilla SGLang lets LC TTFT-p99 (disaggregated) / TBT (co-located) blow past target as BE load grows; SmoothAgent's slack-gated admission keeps LC latency flat — this is the evidence that BE work makes progress without dominating the critical path.



Paper's Figure 15 (caption: "Accuracy of the context-aware performance model from Section 4 under controlled mixed batches with fixed generation work and varying cached-context lengths."). With generation work held fixed, only cached-context length varies; the token-budget-only prediction diverges (41.5% max error) while the context-aware model tracks measured latency (13.7%) — justifying why KV length must enter the admission oracle.

Setup at a glance: H100 80GB / NVLink / EPYC; Qwen3-8B (1 GPU) and Qwen3-32B (TP=4); GQA, 32K max context; 28-step code-analysis task (~600–650 tok/step); baseline = same strategies executed synchronously on SGLang.

---

6. 论证链 #

#	Step	Support (paper-internal)
1	Agent context grows monotonically each turn, causing context rot + cost.	§1–§2; Fig 1
2	Frameworks apply context engineering (offload/reduce/isolate) to cap length.	§1 para 2; Fig 2(a)
3	Each such transform rewrites the prefix → invalidates KV cache → blocking re-prefill → TTFT spike ("transformation overhead").	§1 para 3; Fig 2(b), Fig 3
4	These transforms are segment-decomposable: $T(C)=\	_i T(S_i)$, each segment independent of future tokens.	§3.1; Eq. (segment-decomp)
5	Therefore transforms can run ahead of time in a lookahead stream and be spliced at commit, removing the spike.	§3.2; Fig 4
6	All three strategy families (offloading, reduction, isolation) instantiate the 4-method interface and satisfy the property.	§3.3–§3.5; Figs 5–8
7	Lookahead work risks interfering with LC requests; a context-aware batch-latency model (Eq. 1) predicts co-batch cost including KV length.	§4; Eq. 1; Fig 15
8	A best-effort scheduler admits lookahead only within LC slack (Alg. 1 disaggregated / Alg. 2 co-located), never delaying LC.	§4; Algs 1–2; Fig 14
9	End-to-end this eliminates transform-point spikes: up to 11.9×, avg ~62% co-located / 64.5% disaggregated, 26.8–80.5% across real frameworks.	§6.2; Figs 10–13

---

7. 实现 cross-reference #

Artifact is public: https://github.com/PanZaifeng/SmoothAgent (§0). Specific file:line citations are not resolvable from L1 alone, so concrete internal references are marked [实现未公开] at the line level; the reproducible interface contracts are given verbatim in the paper.

• Strategy interface (should_lookahead / transform / should_commit / should_promote, MainState / LookaheadState) — Listing 1, reproduced verbatim in L1 §3.2. [实现未公开] at file:line; behavior fully specified by the listing.
• Agent-loop hooks (runtime.on_commit, runtime.on_segment_boundary) — Listing 2, L1 §3.2. Two hooks are the entire integration surface.
• Scheduler integrated into SGLang v0.5.9 (§5); Algorithms 1 (PD disaggregated, slack-driven admission) and 2 (PD co-located, priority-ordered decode→LC→BE under TBT $\delta$) reproduced verbatim in L1 §4. KV transfer via NIXL in disaggregation.

核心技术壁垒 (dedicated paragraph) #

The single hardest-to-replicate insight is the isolation streaming reformulation (§3.5). A naïve implementation treats the delegation instruction as one monolithic output and cannot start building the sub-agent context until decoding finishes — so all sub-agent prefill lands synchronously at the switch. SmoothAgent instead partitions the auto-regressive decode itself into fine-grained segments (S2, S3, …), each a chunk of newly generated tokens depending only on the prior prefix, and extends the sub-agent KV cache chunk-by-chunk while the instruction is still decoding. Because instruction decode is inherently slower than prefill, there is enough wall-clock slack to hide nearly all sub-agent prefill, leaving only a short final commit prefill. Replicating this requires (a) making decode emit at segment granularity, (b) preserving the strict prefix-dependency ordering so each lookahead request sees the correct prior state, and (c) sizing segments so the overlap actually hides the prefill without adding decode overhead. This "slowness-as-opportunity" inversion is the paper's subtlest and least obvious engineering.

关键实现细节 (easy-to-miss tricks) #

1. Soft summarization threshold (§3.4 para 7): trigger lookahead summarization at a threshold slightly below the real hard limit (11K vs 15K). Because the exact boundary would otherwise require observing the full context, the soft threshold lets summarization start early; the boundary discrepancy is bounded by the soft–hard gap and, for long contexts, is negligible — and may even improve fidelity by summarizing slightly less. Missing this makes summarization lookahead impossible to start in time.
2. In-order lookahead enqueue (§3.2 para 5): lookahead requests are executed strictly in order so each observes the correct previously-transformed prefix state; and should_promote (default = should_commit) can be overridden to raise scheduling priority before the commit point, reducing commit misses. Getting the ordering wrong silently corrupts the transformed prefix.