Efficient LLM Serving for Agentic Workflows: A Data Systems Perspective — L2 #

1. TL;DR #

Helium serves batch agentic workflows by modeling them as query-plan DAGs with LLM calls as first-class operators, then applies proactive KV/prompt caching + cost-based cache-aware scheduling over a templated radix tree. Up to 1.56× over KVFlow, 0.9% avg gap to MILP optimum, exact semantics.

Agent scope: closed-ish, high-throughput batch exploration (agentic speculation), not interactive single-turn. Interaction is multi-turn but offline-batched — the entire batch of query instances is known at compile time. Autonomy is fully autonomous within a fixed workflow template; the human writes the DSL graph, not the runtime decisions. Simplifying scope (§2): same base LLM for all agents, local data ops only, no remote API calls, on-prem multi-GPU.

2. Q1 / Q2 / Q3 #

Q1 — 痛点 (pain point). Agentic workflows issue many interdependent LLM calls with massive redundancy from overlapping prompts and speculative/parallel exploration. Existing serving stacks split into two myopic pillars: (a) inference engines (vLLM) optimize each call locally with passive, reactive prefix caching and no view of the DAG ("operator-level myopia"); (b) orchestrators (LangGraph, Spark UDFs) treat the LLM as a black box, hiding the stateful KV cache and the bimodal prefill/decode cost from any optimizer. Neither can guarantee cross-call KV reuse when related queries are separated by unrelated ones.

Paper's Figure 1 (caption: "Three disparities between traditional SQL pipelines and agentic workflows with LLM as operators"). The figure grounds the motivation: unlike relational select/filter operators, LLM operators wrap stateful, continuously-batched inference (operator abstraction), carry KV state across calls (inter-operator sharing), and share prompt prefixes across queries and batches (inter-workflow sharing). These three axes are exactly what a workflow-agnostic optimizer cannot exploit.

Q2 — 方法 (method). Helium is a workflow-aware serving layer with a classic three-stage query architecture: (1) a Python lazy-dataflow DSL builds a symbolic DAG; (2) a cost-based, cache-aware query optimizer prunes dead operators, applies common-subgraph elimination (CSE), and substitutes cache-hit operators with lightweight CacheFetch; (3) a query processor builds a templated radix tree (TRT) capturing static+dynamic prompt prefixes and operator dependencies, runs a greedy DFS cache-aware scheduler over a token-step cost model, and pins proactively-precomputed KV caches in vLLM. 核心技术壁垒: the TRT unifies the global prefix hierarchy and the dependency DAG in one structure, enabling proactive scheduling (decisions made at compile time from workload structure) rather than the reactive prefix-matching of SGLang/Parrot that only sees sharing once requests are ready.

Q3 — 结果 (results). Up to 1.56× over KVFlow and 2.21× over Parrot on primitive microbenchmarks; up to 1.34× over KVFlow and 4.25× over OpWise on the end-to-end Trading workflow; 100.92× / 39.50× over the naive query-wise vLLM strawman. The greedy scheduler lands within 0.9% average (3.6% max) of a MILP optimum on scaled-down instances, and its TRT metadata is ~27× smaller than SGLang's RadixCache (552 KiB vs 14.8 MiB at 16 branches). All with bit-exact outputs (deterministic-operator-only caching, greedy sampling).

3. 架构 / 方法图 #

The system is a parse → optimize → process pipeline, with a DSL-defined DAG as input and pinned-KV vLLM workers as the execution substrate.

Paper's Figure 3 (caption: "Overview of Helium's architecture"). Reading left to right: the Agentic Workflow DSL is parsed into a symbolic DAG (TensorFlow-style placeholders, but with cross-operator continuous batching so ready outputs forward without blockage); the logical-plan optimizer rewrites the DAG (prune/merge/CacheFetch) using per-operator profiles (e.g. #input 2482 / #output 470); the processor builds the TRT and schedules operators onto workers to maximize prefix reuse. Note the profiles feed the cost model — this is the "white-box operator" claim made concrete.

The agent loop for a single workflow turn — how one operator moves from optimization to KV-reusing execution — is a plan-then-dispatch cycle rather than a per-turn observe/reflect loop (Helium's autonomy lives in the scheduler, not the LLM):

Memory model. Short-term = each worker's vLLM KV cache (pinned static prefixes + native LRU/longest-prefix eviction for the rest). Long-term/episodic = the global prompt cache mapping deterministic-operator inputs → outputs, persisted across batches (daily-report reuse). The TRT itself is a compile-time index of prefix structure, not a runtime memory.

Error recovery. Two levels: (i) evicted precomputed KV prefixes are simply recomputed next batch (no correctness impact); (ii) the scheduler, if an operator is blocked on data or precedence delay, forces the operator with the earliest start to guarantee progress ("if blocked, force the operator with the earliest start").

Planning & reasoning. Style: predetermined workflow (static DAG) + a critical-path greedy DFS over the TRT — not ReAct or tree search. Decomposition is top-down (workflow template → operators → LLM calls). Budget is implicit: the scheduler minimizes makespan (total token steps), and complexity is bounded by workflow structure, not batch size — the key design choice that keeps scheduling tractable. Backtracking: none at runtime; the "soft schedule" permits reordering to adapt to dynamics, but committed LLM outputs are not undone.

4. 作者证明 #

无形式化作者证明 — 仅实证 for the end-to-end speedup claims. Helium has no success/quality guarantee (it preserves exact semantics by construction, so quality is trivially the base model's). The formal content is instead a scheduling cost model + NP-hardness result + complexity proof + optimality-gap benchmark. There is no convergence bound; what could have been bounded is the makespan approximation ratio of the greedy heuristic (the paper substitutes an empirical MILP gap instead).

Notation table #

Load-bearing equations & physical meaning #

Prefill usage — only new tokens below the LCA with the previous call are charged; the shared prefix is free (this is where prefix-cache reuse enters the cost):

$$u_p(i, j) = \sum_{v \in \text{path}(r,\, l_{ij})} \omega(v) \text{ if } j=1, \text{ else } \sum_{v \in \text{LCApath}(l_{i\,j-1},\, l_{ij})} \omega(v)$$

Decode usage — cumulative token-steps over all generation steps sum to a triangular number (each step re-attends over a growing sequence):

$$u_d(i, j) = \tfrac{1}{2}\, \text{len}_{\text{out}}(l_{ij})\bigl(\text{len}_{\text{out}}(l_{ij}) + 1\bigr)$$

Total — prefill is multiplied by output length because the prefilled prompt is re-read at every decode step:

$$u(i, j) = \alpha_i \bigl(\text{len}_{\text{out}}(l_{ij}) \times u_p(i, j) + u_d(i, j)\bigr)$$

Precedence delay — pushes dependent calls apart so independent calls fill the batch:

$$d(i, j) = \gamma \times \text{len}_{\text{out}}(l_{ij})$$

Makespan objective with dependency + completion constraints:

$$\text{Minimize } T(\sigma) = \max_{i,j} c(i, j) \quad\text{s.t.}\quad b(i,j)\ge c(i',j')+d(i',j'),\ \ c(i,j)=b(i,j)+u(i,j)$$

Optimality gap used to validate the heuristic empirically:

$$\text{Gap (\%)} = \frac{T(\sigma) - T(\sigma^)}{T(\sigma^)} \times 100$$

Six minimum checks #

1. Dimensional consistency: $u_p,u_d,u$ are all in token-steps; $u_d$'s $\tfrac12 n(n+1)$ is token-steps (sum over $n$ decode steps), and $\text{len}_{\text{out}}\times u_p$ correctly turns per-prefill cost into per-decode-step cost. ✓
2. Boundary — no output: $\text{len}_{\text{out}}=0 \Rightarrow u_d=0$ and total prefill weight $\to 0$; a pure-fetch operator costs nothing to decode. ✓ Consistent with CacheFetch being lightweight.
3. Boundary — full prefix reuse: if call $j$'s path shares everything up to its parent with $j{-}1$, $\text{LCApath}$ collapses to the unique suffix; identical calls on the same worker cost only their own leaf tokens. ✓ Matches the SGLang-radix intuition.
4. NP-hardness: reduction from parallel-machine makespan minimization [36]. The dependency + delay constraints are strictly a superset of unconstrained parallel-machine scheduling, so the hardness follows directly. ✓ Plausible.
5. Complexity vs batch size: operator-level TRT gives $O(|V_I|\cdot b^3 + |E'|\cdot h)$ (Appendix B), independent of batch size $N$ — verifiable from Algorithm 1 working on the logical plan, not per-query calls. ✓ This is the central tractability claim.
6. Monotonicity of gap benchmark: greedy gap (0.9% avg) < LSPF (>14%) < Random (~16%) < OpWise (~18%) < QueryWise (72%); ordering matches each method's degree of prefix/parallelism awareness. ✓ Internally consistent (Table 5).

Caveat on check 6: the optimality gap is only measured on 2–4 agent / 2–4 query scaled-down instances with a 6-hour MILP timeout; it does not certify near-optimality at the batch-80 / 16-branch scales used elsewhere.

5. 实验与数据 #

Setup. Qwen3-8B/14B (32B and Llama-3.1-8B in sensitivity), 2×94GB H100 NVL, vLLM v0.16.0 with automatic prefix caching + chunked prefill, greedy sampling throughout. Baselines span both pillars and workflow-aware peers: vLLM (naive query-wise), OpWise (Spark/Dask-style operator-wise), LangGraph, AgentScope, Parrot, KVFlow. Metric is end-to-end wall-clock latency (throughput is its inverse for fixed batch).



Paper's Figure 5 (caption: "Normalized end-to-end latency of Helium and baselines, excluding vLLM, across representative workflows and datasets with Qwen3-8B. ... 1.0 equals the slowest system (lower is better)"). Helium (rightmost bar per group) is lowest in every group. The load-bearing observation: the gap over the strongest baseline KVFlow is largest exactly on high-prefix-sharing workloads (MapRed and Debate with TAT-QA), confirming the mechanism targets prefix redundancy specifically rather than generic parallelism.



Paper's Figure 7 (caption: "End-to-end latency ... on the Trading workflow across batch sizes using (a) Qwen3-8B and (b) Qwen3-14B"). On the realistic 19-agent / 88-operator Trading workflow the KVFlow gap narrows to 1.34× (vs 1.56× on primitives) — the honest signal that composite real workflows dilute the pure prefix-sharing advantage. The advantage widens with batch size, matching the amortization argument (proactive caches pay off over more instances).

The Trading workflow itself is the load-bearing benchmark; its structure explains where each mechanism fires:



Paper's Figure 6 (caption: "The Trading workflow ... combining the Parallel, Debate, and Map-Reduce patterns. Agent annotations indicate opportunities for proactive KV or prompt caching, applied to prefixes over 200 tokens"). Three stages — analyst (Parallel, 4 agents), research (Debate, 2 researchers + manager), decision (MapRed, 8 trader chains × 3 risk agents + fund manager). The annotated >200-token cache opportunities are what CSE and proactive KV target; the near-static fundamentals data is what the cross-batch prompt cache exploits.

Ablation (Trading, bs=16, Qwen3-8B). Latency increase when removing each component: plan pruning 23.35% > cache-aware scheduling 17.66% > prompt caching 13.56% > proactive KV caching 3.55%. Surprising: the headline "proactive KV cache" is the least impactful; the biggest win comes from CSE-driven plan pruning removing redundant (not merely dead) operators.

Scheduling optimality (Table 5). Helium 0.9%±1.4% avg gap (max 3.6%) vs LSPF 14.5% (max 30.5%), Random 16.3%, OpWise 17.6%, QueryWise 72.4% (max 149.2%). Cache-hit-rate isolation (Table 4): Helium 56.5% vs LSPF 37.9% — a 32.9% absolute hit-rate lift from global TRT scheduling over online prefix matching.

Execution dynamics case study confirms the mechanism visually:



Paper's Figure 12 (caption: "Execution dynamics: (a) Batched requests; (b) Effective tokens; (c) Per-request latency CDF. Helium's batch efficiency significantly reduces tail latency"). Helium processes 1.18× more requests per batch and 1.41× higher peak effective tokens, cutting median latency 28.3→20.5 s and p95 51.7→37.2 s. The CDF's tighter tail is the operational payoff of interspersing dependent/independent calls via the precedence-delay model — congestion and stragglers are avoided.

6. 论证链 #

#	Step	Support (paper-internal)
1	Agentic workflows are batch DAGs of stateful, redundant LLM calls, unlike stateless SQL operators	§1 three disparities (Fig 1); §2 workflow patterns (Fig 2)
2	Existing stacks are either operator-myopic (vLLM: passive/reactive prefix cache) or black-box orchestrators (LangGraph/Spark: hide KV + prefill/decode cost)	§2 two pillars
3	Therefore a white-box, workflow-aware layer is needed that does inter-operator + inter-query/batch sharing + a maximizing optimizer, proactively	§1 para 6 "missing jigsaw"; §2 two key ideas
4	Realize it as parse→optimize (prune+CSE+CacheFetch)→process (TRT + cost-based scheduler + pinned KV)	§3 overview (Fig 3); §4 optimizer; §5 processor
5	The TRT + token-step cost model let scheduling be posed as an NP-hard makespan problem, solved greedily with batch-size-independent complexity	§5 formulation & Algorithm 1; App B proof
6	The greedy schedule is empirically near-optimal (0.9% gap) and the full system beats all baselines with exact semantics, ablations attributing gains to pruning/scheduling/caching	§7.5 (Table 5); §7.1–7.3 (Figs 5/7, Table 3)

7. 实现 cross-reference #

Source code referenced by the paper: https://github.com/mlsys-io/helium_demo (footnote, §1) — treat as [实现未公开 in this KB]; only the paper's Listing 1 DSL sketch and prose are available. Key implementation anchors from the text:

• DSL surface (§5 Listing 1 / §6): ops.placeholder, ops.llm, ops.fmt, ops.Msg, graphs.build(...).compile(...), helium.invoke(...). Lazy dataflow — each ops call records a node; graphs.from_ops() traverses terminals to materialize the graph; compile() binds a placeholder to a concrete batch; invoke() triggers rewrite+schedule+dispatch.
• Engine integration (§6): built on vLLM v0.16.0; one vLLM engine per worker in a separate process over IPC message queues. Proactive caching is implemented by augmenting vLLM to pin KV caches of precomputed prefixes — special precompute requests prefill and retain the KV in GPU memory; overflow falls back to vLLM's native block-level LRU + longest-prefix-match eviction (evicted prefixes just recompute next batch).
• Scheduler (§5 Algorithm 1, App A Algorithm 2): operator-level TRT built by topological sort → per-operator prefix template (static tokens + dependency placeholders) → radix insertion with node splitting → dependency edges via node_map. Schedule loops Recurse with a force-progress flag; child selection uses a critical-path heuristic.

关键实现细节 (easy-to-miss tricks):

1. Deterministic-only prompt caching (§4 para 6): the global prompt cache is restricted to non-LLM ops and greedy-sampling LLM ops — this is precisely what makes "exact semantics" defensible, and it is why every experiment uses greedy sampling (a hidden coupling between the correctness guarantee and the eval protocol).
2. Nested-sequence schedule structure (§5 para 14): the scheduler groups operators sharing a static prefix into inner sequences processed query-by-query, but iterates operator-by-operator across inner sequences — reusing per-query prefix inside and static prefix across. A naive operator-by-operator schedule would thrash the cache by alternating queries' unique question prompts; an intermediate buffer releases an inner sequence only once all its static-prefix-sharing operators are collected.

The optimization+scheduling trace makes both tricks concrete on the analyst subgraph:



Paper's Figure 13 (caption: "Helium's optimization and scheduling process: (a) initial DAG, (b) optimized logical plan, (c) TRT construction, and (d) cache-aware schedule"). (a)→(b): the optimizer finds cached fundamentals+social-media outputs and replaces those subgraphs with CacheFetch, pruning two branches. (c): the TRT captures the remaining prefix structure. (d): the scheduler groups Op1/Op2 (market agent) to reuse their shared prefix across the batch, then interleaves the independent news-agent Op4/Op5 to hide Op3's dependency latency before scheduling Op3/Op6 — the precedence-delay incentive from §5 in action.