ThunderAgent: A Simple, Fast and Program-Aware Agentic Inference System #

1. TL;DR #

Agentic inference stacks glue a stateless LLM engine (vLLM) to a stateless tool orchestrator (Kubernetes), scheduling each LLM/tool call independently — causing KV-cache thrashing, cross-node memory imbalance, and leaked tool resources. ThunderAgent makes the whole multi-turn workflow a first-class "agentic program" and schedules at that granularity, yielding 1.5–3.6× serving and 1.8–3.9× RL-rollout throughput.

2. Q1 / Q2 / Q3 #

Q1 — 痛点 (Pain). Modern inference engines lose throughput as the number of concurrent agentic workflows grows, and RL rollout eats >70% of training wall-clock. The root cause is that request-aware components have no end-to-end view of a workflow, producing three concrete pathologies: (1) KV-cache thrashing — the engine evicts a workflow's KV cache during its tool-execution interval with no knowledge it will be reused, forcing a full re-prefill on resume (up to $7.14\times$ latency inflation); (2) cross-node memory imbalance — prefix/KV-aware routers pin all requests with identical agent system prompts to one node, so some DP replicas saturate while others idle (peak 51% imbalance); (3) tool lifecycle obliviousness — orchestrators can't distinguish a temporary tool-wait from termination, so sandboxes/Docker images/ports are never reclaimed and disk grows linearly to exhaustion.

Q2 — 方法 (Method). Abstract each workflow as an agentic program $P = \langle \mathit{ID}, c, \mathcal{T}, \mathcal{L}, \tau, s \rangle$ — a persistent scheduling unit carrying its token count, tool set, node placement, execution phase (Reasoning/Acting), and status (Active/Paused/Terminated). Built on this: (a) a program-aware waiting queue that periodically (every $\Delta t$) detects imminent thrashing and pauses acting programs (whose caches are idle) while restoring reasoning ones, using a time-decay weight $f(t)$ on acting programs and shortest-context-first eviction; (b) a global version of that queue that migrates paused programs to any underutilized replica (recomputation is node-agnostic once paused, so locality is not sacrificed); (c) tool resource management with lifecycle-hook GC (reclaim on Terminated) and asynchronous environment pre-build overlapped with LLM reasoning.

The 核心技术壁垒: the insight that once a program is paused its KV cache is assumed evicted, which makes its recomputation cost node-agnostic — this single observation is what simultaneously breaks the false tradeoff between KV-locality (pin to one node) and load-balance (spread across nodes), letting one global queue solve thrashing and imbalance together. Reproducing this requires wiring program-phase state into both the memory-pressure detector and the cross-node dispatcher coherently; it is not a drop-in engine patch.

Q3 — 结果 (Result). $1.48$–$3.58\times$ serving throughput over vLLM ($1.17$–$3.31\times$ over Continuum) across coding/routing/science agents; $1.79$–$3.92\times$ RL-rollout throughput over vLLM+Gateway on 2×H100; up to $4.2\times$ disk-memory savings; near-100% KV hit rate under high concurrency for deterministic tools. Notably ThunderAgent wins on throughput even when its KV hit rate is lower than Continuum's (stochastic tools), because it prioritizes active GPU utilization over cache pinning.

3. 架构 / 方法图 #

The agent-loop granularity is the whole workflow. One program alternates between a Reasoning phase (on-GPU LLM inference producing thought $\ell_t$ + action $a_t$) and an Acting phase (off-GPU tool execution), accumulating context $c_t = (o_1, e_1, \dots, o_t)$ where each $c_{t+1}$ extends $c_t$ as a prefix — hence near-complete theoretical KV reuse across steps. Planning style is a predetermined ReAct-style think→act loop external to the system (ThunderAgent schedules it, it does not do tree search or backtracking itself). Memory model: short-term = KV cache of $c_t$ in HBM; long-term/environment = persistent tool assets (sandboxes, DB connections, ports) held across the whole trajectory; episodic = the program metadata table. Autonomy is fully autonomous multi-turn; the system optimizes sustained throughput, not tail latency.

Paper's Figure 3. The global waiting queue is queried every $\Delta t$. Backend #1 is thrashing (over capacity) and Backend #3 is underutilized. The scheduler pauses acting Program #2 back into the queue and restores reasoning Programs #6 and #9 onto the idle backend — killing thrashing in #1 and rebalancing #3 in one coordinated action. Notice a single shared queue is what couples the within-node (pause) and cross-node (restore-elsewhere) decisions.

The per-turn state machine and the pause/restore transitions are:

Error recovery is structural rather than semantic: a tool failure or termination surfaces as a status transition to Terminated (triggering GC); a paused program that must resume simply pays a re-prefill on restore. The system does not model tool correctness — it only manages the resource lifecycle around tool calls.

4. 作者证明 #

无形式化作者证明 (success guarantee) — 仅实证 for the end-to-end throughput claims. There is no convergence or success-rate bound; success rate is not even a metric (the system is workload-agnostic and preserves the underlying agent's outputs). What is formally proved is the scheduling-optimality of two mechanisms. What could have been bounded but was not: a worst-case throughput ratio vs. an oracle scheduler.

Notation table

方程物理意义. Cost is the Space-Time Product $\text{Cost}_x = \int_0^{t_x} M_x(t)\,dt$ — area under the memory–time curve. It decomposes as $\text{Cost}_{\text{total}} \approx \text{Cost}_{\text{decode}} + \text{Cost}_{\text{prefill}} + \text{Cost}_{\text{recompute}} + \text{Cost}_{\text{unused}} + \text{Cost}_{\text{caching}}$, where the last three are exactly the three pathologies (thrashing, imbalance, idle-cache-during-tool-wait). The scheduler minimizes those three while maximizing the first two. Thrashing on backend $\mathcal{L}$ is declared when $\text{C}_{\text{total}} < \sum_{p\in\mathcal{L}} c_p$; with the decay, acting-program tokens are discounted: $\text{C}_{\text{total}} < \sum_{p,\tau=R} c_p + \sum_{q,\tau=A} c_q \cdot f(t_q)$. Eviction chooses subset $S$ solving $\min_S \sum_{i\in S} c_i^2$ s.t. $\sum_{i\in S} c_i \ge \Delta C$.

Minimum checks (6):

1. Decay-form derivation. Time-homogeneity $f(t+\Delta)=f(t)f(\Delta)$ + boundary $f(0)=1,\ f(\infty)=0$ is the semigroup/Cauchy equation; log-linearity forces $f(t)=e^{-\lambda t}$ (continuous) or $f(k)=x^{-k},\ x>1$ (discrete). ✔ closed-form consistent.
2. Boundary sanity. $f(0)=1$ ⇒ zero tool time = pure reasoning (no discount); $f(\infty)=0$ ⇒ infinite tool time = request-level (fully evictable). ✔ endpoints match the two baseline regimes.
3. Quadratic recompute. With chunked prefill processing constant KV/iter, $c_i(t)\propto t$, so $\text{Cost}_{\text{recompute}}=\int_0^{t_{\text{rec}}} c_i(t)\,dt \propto t_{\text{rec}}^2$; and $t_{\text{rec}}\propto c_i$ ⇒ $\propto c_i^2$. ✔ dimensionally an area, quadratic in length.
4. Shortest-first optimality. For strictly-convex $x^2$: if $c_{\text{short}} c_{\text{short}}^2 + r^2$, so swapping the long for the short strictly lowers $\sum c_i^2$ while keeping $\sum c_i \ge \Delta C$. ✔ exchange argument valid.
5. Scoring-function consistency. $S_{\text{restore}}=\tfrac{1}{c_P}+\mathbb{I}(\tau=R)$ and $S_{\text{pause}}=\tfrac{1}{c_P}+\mathbb{I}(\tau=A)$: the indicator (0 or 1) dominates the $\tfrac{1}{c_P}\in(0,1)$ term, so phase is the primary key and shortest-context the tiebreak — matches both Lemma 4.1 (evict short) and the intent (pause Acting / restore Reasoning). ✔
6. Node-agnostic recompute claim. Once paused, KV is evicted, so restore cost is a full re-prefill independent of destination node ⇒ $\text{Cost}_{\text{unused}}$ can be driven to zero by dispatching any paused program to any idle replica without extra penalty. ✔ internally consistent with the STP model.

5. 实验与数据 #

Models: GLM-4.6 (MoE), also Qwen-3; TP on H100. Serving on 1×8H100, RL rollout on 2×8H100. Baselines: vLLM + Kubernetes/Gateway, and Continuum (TTL-based KV pinning).



Paper's Figure 1(a). Prior inference systems fail to sustain throughput as the parallel-workflow (batch) count rises — the motivating symptom. This is the curve ThunderAgent is designed to flatten.



Paper's Figure 2(a). Over a 90-minute rollout snapshot across two DP nodes, memory usage diverges >20% for over 37 minutes and peaks at 51% imbalance — direct evidence that prefix-aware routing wastes half of one node's capacity. This is the $\text{Cost}_{\text{unused}}$ term made visible.



Paper's Figure 4. Throughput (steps/min) vs parallel-workflow count across agent–benchmark pairs. ThunderAgent ($1.48$–$3.58\times$ over vLLM) holds near-flat maximum throughput past the point where baselines collapse once workload exceeds GPU-memory limits, and adapts to available capacity without manual tuning.



Paper's Figure 1(c). Head-to-head speedup for SWE-Agent, OpenHands, and ToolOrchestra: the gain grows with concurrency, confirming that reduced KV-thrashing plus tool-lifecycle management is what buys sustained throughput rather than a fixed constant offset.



Paper's Figure 6. (a) End-to-end latency breakdown for OpenHands rollout: gains come primarily from prefill+decode reductions, while tool-resource management adds only ~10% latency improvement yet delivers the $4.2\times$ disk savings. (b) Sensitivity to detection period $\Delta t$ and decay base $x$ in $f(t)=x^{-t}$: throughput is robust across settings; larger $\Delta t$ risks thrashing between checks, larger $x$ trades recomputation for reduced caching cost.

RL rollout (Table 2, 2×H100, N=144, 3 h):

Workflow	System	Throughput
mini-SWEAgent	vLLM + Gateway	375.4
mini-SWEAgent	ThunderAgent	671.8 ($1.79\times$)
OpenHands	vLLM + Gateway	69.1
OpenHands	ThunderAgent	270.8 ($3.92\times$)

Benchmarks: SWE-Bench / SWE-Bench Lite (coding), HLE-Bench (routing, ToolOrchestra), ScienceAgentBench (science). Metrics are throughput (steps/min), KV-cache hit rate, and end-to-end latency — not task success (the system is output-preserving). The most instructive result is the counter-intuitive one: in stochastic-tool settings Continuum reaches a higher KV hit rate yet lower throughput, because pinning caches for long tool calls inflates $\text{Cost}_{\text{caching}}$ — so hit rate is not a monotone proxy for throughput.

6. 论证链 #

#	Step	Support
1	Agentic workflows have near-complete theoretical KV reuse across steps ($c_{t+1}\supseteq c_t$) and hold persistent tool state.	§2.1 formal model
2	Request-level schedulers ignore this: they evict KV during tool waits and never reclaim tool resources, so throughput collapses (7.14× latency, 51% imbalance, linear disk growth).	§3 profiling, Fig 1a/1b/2
3	Therefore the correct scheduling unit is the whole workflow; encode it as a program tuple $P$ carrying phase and status.	§4.1 abstraction
4	Cast the objective as STP-cost minimization; the three pathologies map to $\text{Cost}_{\text{recompute}}, \text{Cost}_{\text{unused}}, \text{Cost}_{\text{caching}}$.	§4.2 cost model
5	Minimize those terms: pause Acting programs (shortest-first, decay-weighted) to stop thrashing; because paused-cost is node-agnostic, a global queue can also rebalance nodes; hook-GC + async prep kill the tool-resource terms.	§4.3–§4.4, Thm E.1, Lemma 4.1
6	Result: sustained throughput that does not collapse past memory limits — 1.48–3.58× serving, 1.79–3.92× rollout, 4.2× disk.	§5 experiments, Table 2, Fig 4/6

7. 实现 cross-reference #

Open-sourced at github.com/Agentic-Kinetics/ThunderAgent (specific file:line not extracted in L1) → treat concrete internals as [实现未公开 at file:line granularity], but the interface contract is fully specified.

• Integration surface (Appendix B.3, Fig 8): adoption is three changes — attach program_id to each LLM request, attach program_id to each tool execution, and send an explicit release signal with program_id on program end. All other OpenAI-style API fields are unchanged. ThunderAgent wraps existing engines/orchestrators as a middleware runtime layer.
• State schema (Appendix B.1, Tables 3–4): ProgramState{status: ProgramStatus, backend_url: str, step_count: int, total_tokens: int}; ProgramStatus ∈ {REASONING (on-GPU), ACTING (off-GPU tool), PAUSED (global paused set), STOPPED (reclaimed)}; BackendState{url, healthy: bool, cache_config: Optional[CacheConfig], active_program_tokens: int}.
• 核心技术壁垒 (§4.3.2): the load-bearing engineering trick is that pausing marks the KV as evicted, decoupling recomputation cost from placement. This is what lets a single global waiting queue serve both thrashing-prevention (which program to pause on an over-full node) and load-balancing (which idle node to restore onto) without a separate KV-migration path over the interconnect — the thing distributed-KV systems (BanaServe, LMCache) pay bandwidth for, ThunderAgent sidesteps by recomputing.

关键实现细节 (easy-to-miss):

1. Both watermarks $\lambda_{\max}=\lambda_{\min}=1$ (§4.3.1) — no explicit hysteresis margin; the shared agent system prompt implicitly reserves buffer. Fragile for heterogeneous-prompt workloads; no analysis of the breaking point.
2. Tool env preparation is triggered by the queue, not the LLM — when a high-priority program nears the restore threshold, its Docker/deps are pre-built asynchronously before GPU memory is granted (§4.4), hiding init latency; this coupling of the tool-orchestrator prefetch to the GPU-scheduler's lookahead is what yields the ~10% latency slice plus 4.2× disk savings.
3. Production posture: sandboxing via Docker containers reclaimed by reference-count on program termination; observability via the central program metadata table; tool-time is treated as heavy-tailed/unpredictable (Appendix C, Fig 9), which is precisely why static TTL (Continuum) is rejected in favor of the memoryless-decay $f(t)$.