Concur: Proactive Agent-Level Admission Control for Efficient Agentic Batch Inference
Concur: Proactive Agent-Level Admission Control for Efficient Agentic Batch Inference #
Qiaoling Chen, Zhisheng Ye, Tian Tang, Peng Sun, Boyu Tian, Guoteng Wang, Shenggui Li, Yonggang Wen, Zhenhua Han, Tianwei Zhang | 2026-01 | https://arxiv.org/abs/2601.22705 Category: framework | Tags: kv-cache, admission-control, agentic-inference, scheduling, congestion-control, aimd Read: 2026-04-18
Core Contribution #
Concur reframes GPU KV-cache as a shared, finite resource (analogous to network bandwidth) and introduces an AIMD-inspired, agent-level admission controller that proactively regulates the number of concurrent agents based on cache usage and hit-rate feedback, eliminating "middle-phase thrashing" in offline agentic batch inference and delivering up to 4.09× throughput on Qwen3-32B and 1.90× on DeepSeek-V3 without changes to the serving engine.
Summary #
Motivation. Agentic batch inference (RL rollouts, data distillation, agent-as-judge evaluation) is fundamentally different from chat serving: an agent executes a long-horizon ReAct loop, its context grows monotonically, and it interleaves generation with tool calls. While one agent is blocked on a tool, its prefix sits idle in GPU memory. Under the standard request-level LRU eviction used by vLLM / SGLang, those idle-but-semantically-critical prefixes lose recency and get evicted first. When the agent resumes, the engine must recompute or PCIe-reload the entire prefix — and this cost is paid repeatedly as agents oscillate between generation and pausing. The authors characterize this as middle-phase thrashing: a 3-phase execution pattern (warmup → thrashing → cooldown) where KV-cache usage stays pinned near 100% but hit rate collapses and recomputation consumes ~49% of end-to-end latency. Counterintuitively, adding more agents reduces throughput in this phase.
Method. Concur argues that the fix is not another eviction heuristic but a paradigm shift: from reactive request-level cache management to proactive agent-level admission control. It sits as a lightweight middleware between the agent execution framework and the LLM engine (SGLang), exposing three primitives — admit, pause, resume — that act on agents, not requests. The core algorithm reinterprets TCP AIMD congestion control: the "congestion window" $W_t$ is the number of active agents, "packet loss" is cache eviction, and "retransmission" is KV-cache recomputation. Two runtime signals drive the controller: KV-cache usage $U_t$ (proactive) and cache hit rate $H_t$ (reactive). The control law is:
$$W_{t+1} = \begin{cases} W_t + \alpha & \text{if } U_t < U_{low} \\ W_t - \beta W_t & \text{if } U_t > U_{high} \land H_t < H_{thresh} \\ W_t & \text{otherwise} \end{cases}$$
with standard AIMD values $\alpha=2$, $\beta=0.5$, $U_{low}=0.2$, $U_{high}=0.5$, $H_{thresh}=0.2$.
Results. On H100 NVLink clusters, Concur achieves 4.09× speedup over vanilla SGLang at Qwen3-32B/Batch-256/TP-8, and 1.68×–1.90× on DeepSeek-V3 at batch-16–40/TP-16. Against a request-level admission baseline and HiCache (CPU offloading), Concur wins because it preserves agent-level memory locality: it admits agents that have a cache to keep hot, rather than letting reactive eviction churn. KV-cache hit rate stays at 73–96% where vanilla SGLang collapses to 35%.
Key Findings #
- Middle-phase thrashing is a new, 3-phase performance pathology distinct from classical thrashing — memory stays full but hit rate collapses, consuming 90%+ of execution time and 49% of latency as pure recomputation overhead.
- Request-level admission control is insufficient for agent workloads — limiting in-flight requests still lets the cache footprint balloon because it ignores agent continuity (the request-level baseline can be worse than uncontrolled SGLang, e.g. 0.72× in one Qwen3-32B config).
- Cache offloading (HiCache) alone doesn't help at scale — at high concurrency, PCIe contention dominates and offload latency exceeds recomputation. HiCache hits 97% cache hit rate but is 0.34×–0.54× slower than SGLang on DeepSeek-V3.
- AIMD on KV-cache works — the TCP analogy transfers cleanly: KV-cache = bandwidth, eviction = packet loss, recomputation = retransmission. A single fixed parameter set works across all models and workloads.
- Static admission limits are brittle — fixed concurrency (30/32/64/128) either underutilizes GPUs or triggers thrashing; the adaptive policy beats the best static choice by 1.5–2.9×.
- $U_{low}$ is more sensitive than $U_{high}$ — $U_{high}$ can range 0.5–0.6 safely, but $U_{low}$ must be 0.2; too-low $U_{low}$ permanently starves the window, too-high $U_{low}$ over-admits.
Key Figures #
Figure 1: KV-cache growth & offload vs. recompute latency (Motivation) #
What it shows: (a)(b) Input length and KV-cache memory consumption grow monotonically over 10 generation steps for DeepSeek-V3 and Qwen3-32B — a context explodes from a few thousand tokens to tens of thousands within a single agent. (c) GPU→CPU KV-cache offload latency becomes worse than prefill-based recomputation as concurrency increases, for a DeepSeek-V3 request with 6.67 GB cache / 4096 tokens.
Why it matters: Establishes two foundations: (1) agent KV-cache is a growing, long-lived resource, not a short-lived optimization; (2) PCIe offloading — the standard escape valve — fails under the very high-concurrency regimes that agent batch inference targets. This forces the authors to reject a "bigger swap pool" solution and motivates a bandwidth-style admission approach.
Detailed description: The left two panels (a, b) are staircase plots where each step increases input length and KV footprint by roughly a thousand tokens; by step 10 a single agent holds multi-GB of cache. The right panel (c) is a latency comparison across concurrency levels showing offload latency (orange) rising sharply while recompute latency (blue) stays flatter — the crossover point is exactly the regime where batch inference operates.
Figure 2: Three-agent thrashing example vs. agent-level control #
What it shows: A simplified 3-agent timeline contrasting two policies. (a) Without admission control: A1 and A2 pause for tools, A3 keeps generating, LRU evicts A1/A2's prefixes; when A1/A2 resume, both must recompute their entire history, then A3 gets evicted in turn. (b) With Concur's agent-level admission: only 2 agents (A1, A2) are admitted; A3 waits in a pending queue until A2 completes and releases its cache, at which point A3 is promoted. No eviction, no recomputation.
Why it matters: This figure is the visual crux of the paper — it shows why reactive request-level eviction fundamentally cannot win: the problem is not which entry to evict but that too many agents are admitted in the first place. The contrast is the "aha moment" for the AIMD analogy.
Detailed description: Top half (a) shows a Gantt-like timeline with colored bars per agent; red hatched regions mark recomputation and black boxes mark eviction events — they cluster in the middle phase. Bottom half (b) uses the same layout but A3 is grayed out (pending) until A2 finishes; no red hatching appears, and the total timeline is visibly shorter.
Figure 3: Three-phase execution pattern (empirical characterization) #
What it shows: Time-series from a real DeepSeek-V3 agent deployment. (a) Aggregate KV-cache usage (bottom) and cache hit rate (top) over wall-clock time — with yellow/red/green shaded regions for the warmup, middle (thrashing), cooldown phases. (b) Latency breakdown: prefill / decode / recomputation / tool-call time across the three phases. Recomputation is 49.1% of middle-phase latency.
Why it matters: This is the empirical evidence that middle-phase thrashing is not a toy observation but dominates >90% of execution time. The hit-rate curve (high → collapse → partial recovery) is the signature pathology the paper names and fights.
Detailed description: (a) KV-cache usage climbs to ~90% in warmup and stays pinned there through the middle phase — but the hit-rate line plummets from ~90% at the end of warmup to ~30% and stays flat; only in cooldown (after some agents finish) does hit rate recover. (b) Horizontal stacked bars showing that in the middle phase, the "extra recomputation" segment becomes comparable in size to prefill+decode combined.
Figure 4: System overview #
What it shows: Three-layer architecture — Agent Execution Layer (top, multiple ReAct-loop agents) → Agent-Level Controller (middle, the Concur middleware) → LLM Serving Engine (bottom, e.g. SGLang). Arrows show the 4-step workflow: ① agent submits generation request to controller → ② admitted agents reach the engine → ③ tool calls pause agents → ④ controller reads runtime feedback (usage, hit rate) and updates the admission policy.
Why it matters: Shows Concur is non-intrusive: it doesn't replace SGLang, doesn't change KV-cache layout, doesn't need custom kernels. It's a control layer. This is the deployment story that makes the work practically adoptable.
Detailed description: Clean block diagram with the controller box expanded to reveal admit/pause/resume primitives and the AIMD state machine. Bidirectional arrows between controller and engine carry the two feedback signals ($U_t$, $H_t$); the admission decision flows upward to agents in the form of ADMIT/PAUSE tokens.
Figure 5: Temporal KV-cache dynamics — Qwen3-32B Batch-256 TP-2 #
What it shows: Two time-series plots for a hard configuration (Qwen3-32B, batch 256, only 2 GPUs via TP=2). Top: cache hit rate over time. Bottom: cache usage over time. Blue = SGLang baseline, orange = Concur.
Why it matters: Concrete proof that Concur stabilizes the thrashing regime. Under identical conditions, the baseline's hit rate collapses into the 30–50% range while Concur holds 70–90%; usage is bounded just under capacity rather than pinned at 100%.
Detailed description: The top panel shows a dramatic divergence after an initial shared warmup — baseline hit rate oscillates wildly downward, Concur's stays high and much smoother. The bottom usage panel shows the baseline pinned flat at the capacity ceiling, while Concur hovers at ~80% with small AIMD-driven oscillations (the sawtooth signature of additive-increase / multiplicative-decrease).
Figure 6: Static vs. adaptive admission control #
What it shows: Bar chart of end-to-end latency for Qwen3-32B Batch-256 TP-2 under fixed admission levels (30, 32, 64, 128) versus Concur's adaptive policy. Concur is the shortest bar at 846 ms.
Why it matters: Closes the loop on the Q3 research question: no fixed concurrency can match adaptive control, because the workload changes regime across the three phases — a level that's safe during warmup triggers thrashing in the middle phase.
Detailed description: Small fixed levels (30, 32) win modestly over uncontrolled SGLang but leave GPUs idle; the 128 level is slower than 64 due to thrashing; Concur beats the best static choice (64) by 1.5× and the worst (128) by 2.9×.
Key Tables #
Table 1: End-to-end latency and speedup under varying concurrency #
| Model | Batch / TP / #GPU | SGLang (s) | SGLang w/ Request Ctrl (s) | SGLang w/ HiCache (s) | Concur (s) |
|---|---|---|---|---|---|
| Qwen3-32B | 256 / 8 / 8 | 1480 (1.00×) | 2049 (0.72×) | 976 (1.52×) | 362 (4.09×) |
| Qwen3-32B | 256 / 4 / 4 | 2213 (1.00×) | 1089 (2.03×) | 1678 (1.32×) | 757 (2.92×) |
| Qwen3-32B | 256 / 2 / 2 | 2527 (1.00×) | 1383 (1.83×) | 1112 (2.27×) | 846 (2.99×) |
| DeepSeek-V3 | 16 / 16 / 16 | 873 (1.00×) | 861 (1.01×) | 2559 (0.34×) | 521 (1.68×) |
| DeepSeek-V3 | 32 / 16 / 16 | 1226 (1.00×) | 1367 (0.90×) | 2277 (0.54×) | 1018 (1.20×) |
| DeepSeek-V3 | 40 / 16 / 16 | 3877 (1.00×) | 2903 (1.34×) | 2320 (1.67×) | 2043 (1.90×) |
Takeaway: Concur is the only system that wins across all 6 configurations. Request-level control can hurt (0.72× on Qwen3-32B Batch-256 TP-8), and HiCache can be catastrophic on DeepSeek-V3 (0.34×) because PCIe becomes the bottleneck. Concur's margin widens as per-GPU concurrency grows (TP decreasing or batch increasing) — exactly where thrashing is worst.
Table 2: KV-cache hit rate (%) — DeepSeek-V3, TP=8 #
| Batch | SGLang | HiCache | Request Ctrl | Concur |
|---|---|---|---|---|
| 16 | 80.38 | 97.48 | 69.00 | 96.38 |
| 32 | 77.72 | 97.13 | 68.39 | 93.65 |
| 40 | 35.41 | 96.08 | 32.21 | 73.36 |
Takeaway: HiCache wins the raw hit-rate metric (CPU offload keeps everything) but loses on latency (PCIe is slow). Concur gets ~94% of HiCache's hit rate without offloading — proving that keeping cache on-GPU via admission control is the right lever, not moving bytes across PCIe.
Table 3: Sensitivity to $U_{low}$ and $U_{high}$ (Qwen3-32B latency in ms) #
| Varying $U_{high}$ ($U_{low}=0.2$) | TP8 | TP4 | TP2 | Varying $U_{low}$ ($U_{high}=0.5$) | TP8 | TP4 | TP2 |
|---|---|---|---|---|---|---|---|
| 0.4 | 529 | 1089 | 1390 | 0.1 | 2894 | 2561 | 972 |
| 0.5 | 362 | 757 | 846 | 0.2 | 362 | 757 | 846 |
| 0.6 | 386 | 775 | 945 | 0.3 | 779 | 1566 | 1008 |
| 0.8 | 1898 | 2300 | 2498 | 0.5 | 2763 | 2245 | 1451 |
Takeaway: $U_{high}$ is robust in [0.5, 0.6]; $U_{low}$ is narrower — both too-low and too-high values cost 2–8×. The controller's "probing" semantics (additive increase when $U_t 时代定位: 2025年底，LLM inference 优化的"传统红利"（continuous batching, paged attention, chunked prefill, prefix caching）已被 vLLM/SGLang/TRT-LLM 充分吸收。下一代瓶颈来自 agentic workload 的质变——单请求语义消失，被长时序、带外部状态的 agent trajectory 取代。Concur 代表了一个明确的转折：从"请求调度"的维度转向"agent 生命周期调度"，这是 post-2025 agent-native 推理系统的第一波范式级工作之一（与 TokenCake、Continuum、Kairos 同期）。 为何不可 X？(约束推导) 核心技术壁垒: 论文的核心壁垒不是 AIMD 算法本身（这是 30 年老算法），而是把"KV-cache usage"与"hit rate"两个信号联合使用作为触发条件： $$W_{t+1} = W_t - \beta W_t \iff U_t > U_{high} \land H_t < H_{thresh}$$ 为什么需要合取？因为高 usage ≠ 拥塞：在 warmup 末期 usage 可能已经 ~80%，但 hit rate 仍然 90%（cache 是"有用地满"）。只有当 hit rate 也塌方时，系统才真正处于 thrashing 状态。单用 usage 作信号会在健康的高负载时错误减窗（对应 Table 3 中 $U_{high}=0.4$ 的 4–5× 退化）。这个"usage 代表潜在拥塞、hit rate 代表真实拥塞"的双信号设计是论文能 work 的核心工程洞察。 质疑假设: 方法严重依赖 "hit rate 是 thrashing 的可靠信号"。这在有共享 system prompt 的 agent 工作负载下成立（warmup 期 hit rate ~90%），但在无共享前缀的场景（例如每个 agent 处理独立 task、prompt 高度多样化）下 hit rate 可能从一开始就很低，导致控制器无法区分"冷启动"和"thrashing"，进而永久卡在小窗口。论文没有在这类 workload 上评测。笔者不完全确定 Concur 在 heterogeneous workload mix（短 agent + 长 agent）下是否仍有效——公平性可能变成新问题。 设计绑定批判: 生态影响追踪: 本文是 2026-01 刚上传，具体下游采纳尚未发生，但可以预测：(1) SGLang 的下一代 See Figure 4 above. Control-plane vs data-plane separation: Concur is pure control plane. All data (tokens, KV cache, logits) flows through the unchanged SGLang data plane. The controller only exchanges small metadata: (a) admission decisions (admit/pause/resume messages, O(agents)) and (b) feedback signals (usage %, hit rate %, O(1) per polling tick). Stateful: agent execution state (prompt, intermediate steps, tool results) lives in the agent layer; KV cache lives in SGLang; controller state is just $W_t$ + pending queue. Failure handling: Not directly addressed. If controller crashes, fall back to "admit all" (same as uncontrolled SGLang). 3a. Alternative approaches considered (and why rejected): 3c. Assumption audit: 3d. Core technical barrier (re-stated): dual-signal ($U_t \land H_t$) gating. See Phase 2. 3e. Design bindings: Primary bottleneck: scheduling-bound (admission policy) during middle phase; converts to compute-bound once Concur removes the recompute overhead. (See the main "Infrastructure Impact" section above.)Limitations #
Infrastructure Impact #
Deep Analysis (framework) #
Inherited Phase 2 — Constraint Derivation & Technical Barrier #
RadixAttention scheduler 很可能引入 agent-level session 概念；(2) OpenHands / AgentScope 等 agent framework 会适配这类 pause/resume 接口；(3) InternEvo、verl、Nemo-RL 等 agentic-RL 训练栈会把 Concur-style 控制接入 rollout engine——这正是 Peng Sun、Tianwei Zhang 等作者在上海 AI Lab / SJTU 的上下游工作。
1. System Scope #
2. Architecture & Data Flow #
2a. End-to-End Data Flow #
Stage Input → Output Location Latency Data format Agent step gen Tool result → next request Agent runtime (CPU) variable JSON/text Admission check request + (W_t, U_t, H_t) → admit/pause Controller (CPU) <1 ms metadata Prefill (if cache miss) tokens → KV cache GPU HBM 10–500 ms [L, H, S, D] Decode KV + input → token GPU HBM 10–50 ms/tok [V] Tool call text → tool output CPU/external 50–5000 ms text 2b. Data Movement Hotspots #
3. Design Space & Constraint Analysis (extends Phase 2) #
Alternative Feasible? Why fails Bigger KV pool / more HBM ❌ KV grows monotonically with agent steps; any pool saturates Smarter eviction (LRU-k, ARC, TTL) ❌ Inactive-but-live prefixes always rank low by recency/frequency CPU offloading (HiCache) ❌ at scale PCIe contention dominates; Fig 1(c), Tab 1 Request-level admission ❌ No visibility into agent-cumulative cache Static agent-level cap ❌ Workload phase changes — no single limit works (Fig 6) Adaptive agent-level AIMD ✅ This paper Predictive (ML-based) admission ? Untried; potentially higher overhead
4. Key Innovations #
Innovation Mechanism Benefit Cost/Tradeoff Middle-phase thrashing identification Empirical 3-phase characterization (Fig 3) Defines a new category of perf pathology; motivates agent-level thinking Requires running long-horizon workloads to observe Agent-level admission (vs request-level) Treat agent as scheduling unit with pause/resume Preserves cache locality; works with existing engines Forces coherent agent-step boundaries AIMD on KV-cache Map cwnd↔active agents, loss↔eviction, retrans↔recompute 30 years of TCP wisdom transfers for free; no tuning Bound to hit-rate signal quality Dual-signal congestion detection Decrease window iff $U_t>U_{high} \land H_t Avoids false-positive throttling at healthy high-load Adds $H_{thresh}$ hyperparam Non-intrusive middleware Pure control layer above SGLang Zero engine modification; adoption-friendly Needs engine to expose usage/hit-rate metrics 5. Scheduling & Resource Management #
6. Target Scenarios #
Scenario Pattern Goal Why existing fails Agentic RL rollout 100s of agents × ReAct loop, shared system prompt Minimize batch completion Long prefix → middle-phase thrashing Data distillation from reasoning model CoT + tool, variable length Throughput Same Agent-as-judge (multi-scenario eval) 1000s of short-ish agents Max parallelism Cache churn when scaled up 7. Performance Evaluation #
7a. Metrics #
Metric Definition Direction End-to-end batch latency Wall-clock to complete all agents in batch ↓ better Speedup (×) baseline latency / system latency ↑ better KV-cache hit rate (cache-hit tokens) / (total prefix tokens queried) ↑ better KV-cache usage (used slots) / (total slots) Target: balanced, not extreme 7b. Before-after (from Tab 1) #
Config Baseline SGLang Concur Speedup Conditions Qwen3-32B, B256 TP8 1480 s 362 s 4.09× 8× H100 Qwen3-32B, B256 TP2 2527 s 846 s 2.99× 2× H100, most-constrained DeepSeek-V3, B40 TP16 3877 s 2043 s 1.90× 16× H100 7c. Bottleneck shift #
Before: KV-cache-thrashing-bound (49% recompute)
→ with Concur: compute-bound (prefill + decode)
→ remaining bottleneck: intrinsic KV growth (still need cache compression, MLA, etc.)
7d. Baselines & fairness #
8. API & Usability #
admit/pause/resume to agent frameworks; consumes engine's usage/hit-rate metrics (SGLang's /metrics endpoint).9. Infrastructure Impact #
10. Comparison Matrix #
Feature Concur vLLM SGLang TRT-LLM HiCache Continuous batching ✓ (inherits) ✓ ✓ ✓ ✓ Paged attention ✓ (inherits) ✓ ✓ ✓ ✓ Agent-level admission ✓ (unique) ✗ ✗ ✗ ✗ AIMD cache control ✓ ✗ ✗ ✗ ✗ Prefix caching ✓ (inherits) ✓ ✓ (Radix) ✓ ✓ CPU KV offload orthogonal ✓ (LMCache) ✓ (HiCache) partial ✓ Speculative decoding orthogonal ✓ ✓ ✓ ✗ Multi-node ✓ (inherits) ✓ ✓ ✓ ✓ 11. Adoption, Maturity & Ecosystem Influence #
Open Questions #
Related Papers (in knowledge base) #