Splitwise: Efficient Generative LLM Inference Using Phase Splitting
Splitwise: Efficient Generative LLM Inference Using Phase Splitting — L2 #
1. TL;DR #
LLM inference has two phases with opposite hardware profiles: prompt (compute-bound) and token generation (memory-bound, power-insensitive). Splitwise runs them on separate, phase-matched machine pools, transferring the KV-cache over InfiniBand with layer-wise async overlap. Result: up to 1.4× throughput at 20% lower cost, or 2.35× throughput at iso-cost/power.
2. Q1 / Q2 / Q3 #
Q1 — 痛点 (Pain point) #
Running prompt and token generation on the same machine forces a single hardware choice onto two workloads with contradictory demands. The prompt phase runs all input tokens through one parallel forward pass, saturating GPU FLOPs. The token generation phase produces one token per forward pass using the KV-cache, leaving compute idle while being bound by memory bandwidth and capacity. Table I quantifies why this hurts: from A100→H100 compute grew $3.43\times$ but HBM bandwidth only $1.64\times$ and capacity $1.00\times$ — so buying the latest GPU over-pays for compute the token phase cannot use. Mixed batching of the two phases on one machine also causes token latency (TBT) tail inflation when large prompts land in the same batch, forcing SLO over-provisioning.
Q2 — 方法 (Method) #
Split the two phases onto separate machine pools and manage each with hardware matched to its bottleneck. Splitwise maintains a prompt pool, a token pool, and an elastic mixed pool; a two-level scheduler (cluster-level CLS for pool management + JSQ routing; machine-level MLS for batching) assigns each request a prompt+token machine pair. The prompt machine generates the first token and streams the KV-cache to the token machine, which continues generation. The 核心技术壁垒 is making the KV-cache transfer nearly invisible: a layer-wise asynchronous transfer overlaps each layer's KV-cache shipment with the prompt computation of the next layer, over back-plane InfiniBand using the MSCCL++ zero-copy one-sided put primitive — so the transfer cost hides behind compute that is happening anyway (Q2 core barrier expanded in §7).
Q3 — 结果 (Results) #
On production Azure coding + conversation traces with BLOOM-176B and Llama2-70B: KV-cache transfer overhead is $<7\%$ of prompt-compute time and only $0.8\%$ of E2E; it adds $16.5\%$ to the second-token latency vs $64\%$ for a naive serialized transfer. At the cluster level, Splitwise-AA gives $1.4\times$ throughput at $20\%$ lower cost vs Baseline-H100; other variants reach $2.35\times$ throughput at iso-cost/power, or Splitwise-HHcap matches Baseline-H100 throughput at $25\%$ lower power. Performance model MAPE is $<3\%$.
3. 架构 / 方法图 (Architecture) #
Paper's Figure 10, verbatim (caption: "High-level system diagram of Splitwise").
Figure 10 shows the control plane: the cluster-level scheduler (CLS, ①) manages the three machine pools (③④) and routes each request via Join-the-Shortest-Queue, while the machine-level scheduler (MLS, ②) on each node owns its pending queue and per-iteration batching. The key edge is ⑤ — the KV-cache transfer from a prompt machine to its paired token machine. Note the scheduler assigns both machines simultaneously so the transfer can overlap with prompt compute.
Paper's Figure 1, verbatim (caption: "An LLM inference example").
Figure 1 is the conceptual foundation: the prompt phase consumes all input tokens in one pass to emit the first token and populate the KV-cache, then the token phase loops one-token-at-a-time reading that cache. This is the two-phase asymmetry the entire design exploits.
The prompt/token/mixed pool interaction can be summarized as a state machine:
FCFS, ≤2048 tok batch] CLS -->|assign pair| T[Token machine
FCFS, mem-aware batch] P -->|layer-wise KV-cache
over InfiniBand| T P -.->|queue too long| MIX[Mixed pool
mixed continuous batching] T -.->|queue too long| MIX MIX -.->|queue drained| P MIX -.->|queue drained| T
Machines temporarily borrowed into the mixed pool keep their home identity and return once their opposite-kind queue drains; infrequent load shifts trigger coarse re-purposing between pools.
4. 作者证明 (Author's derivation) #
无形式化作者证明 — 仅实证。The paper has 0 numbered display equations; its claims rest on characterization and simulation rather than closed-form proof. The load-bearing quantitative arguments and their internal checks:
| # | Check | Basis | Verdict |
|---|---|---|---|
| 1 | Compute scales faster than memory across GPU generations | Table I: TFLOPs $3.43\times$, HBM BW $1.64\times$, capacity $1.00\times$ | Consistent — motivates memory-cheap token hardware |
| 2 | Token phase runs with very few active tokens | Fig 4: conversation spends 60–70% of time with $\leq$20 active tokens; coding >20% of time at 1 token | Consistent with per-request one-token-at-a-time generation |
| 3 | E2E is token-dominated even for prompt-heavy coding | Fig 5(c): BLOOM-176B 1500-token prompt ≈ time of 6 output tokens | Internally consistent (prompt is one pass; tokens are serial) |
| 4 | Token phase is power-insensitive | Fig 9: 700→350W (>50% cap) yields ~no token latency change; prompt highly sensitive | Consistent — grounds Splitwise-HHcap |
| 5 | Prompt throughput saturates past 2048 tokens; token throughput scales to OOM | Fig 6(a) drop after 2048; Fig 6(b) rises until batch 64 OOM | Consistent — sets MLS 2048-token prompt cap |
| 6 | Transfer overhead is small and mostly hidden | Fig 14/15: $<7\%$ prompt-compute, $0.8\%$ E2E, 16.5% second-token | Consistent — validates the async layer-wise design |
Capacity budget (first-principles, from Table III). KV-cache bytes per token $\approx 2 \times n_{\text{layers}} \times d_{\text{model}} \times \text{bytes}$ (K and V), before GQA reduction. For BLOOM-176B ($70$ layers, hidden $14336$) at FP16: $2 \times 70 \times 14336 \times 2 \approx 4.0$ MB/token; for Llama2-70B ($80$ layers, hidden $8192$): $2 \times 80 \times 8192 \times 2 \approx 2.6$ MB/token (dense-attention upper bound; Llama2-70B's GQA cuts this substantially). This scale explains why the KV-cache transfer (proportional to prompt length) is the dominant Splitwise overhead and why hiding it behind layer compute matters.
5. 实验与数据 (Experiments & data) #
Paper's Figure 3, verbatim (caption: "Distribution for prompt and generated tokens").
Figure 3 grounds the workload asymmetry: coding has a large median prompt (1500 tokens) but tiny median output (13 tokens), while conversation is near-bimodal with median output 129. The reader should notice the ~115× input:output imbalance for coding — the token pool for coding can therefore be small.
Paper's Figure 6, verbatim (caption: "Impact of batching on the throughput for the 2 LLMs").
Figure 6 is the direct source of two MLS policies: prompt throughput (a) degrades past 2048 tokens (median prompt already fills a batch), so the MLS caps prompt batches at 2048; token throughput (b) climbs monotonically with batch size until OOM, so the MLS batches tokens as aggressively as memory allows.
Paper's Figure 9, verbatim (caption: "Impact of power cap on the prompt and token generation latency with the maximum batch size possible").
Figure 9 is the empirical basis for Splitwise-HHcap: prompt latency (a) rises sharply as the power cap tightens, but token latency (b) is essentially flat down to a 50%-per-GPU cap. This asymmetry lets token machines run at 70% rated power with no throughput loss.
Paper's Figure 14, verbatim (caption: "Overhead of the KV-cache transfer as the prompt size increases on A100s and H100s").
Figure 14 shows the serialized transfer growing linearly with prompt size, while the optimized layer-wise transfer flattens to a constant ~8ms (A100) / ~5ms (H100) non-overlapped tail — the H100 setup being ~2× faster tracks its 2× InfiniBand bandwidth. Splitwise picks serialized for small prompts (<512 on H100) and layer-wise for large.
Paper's Figure 18, verbatim (caption: "Summary of throughput-optimized cluster designs"; (a) iso-power, (b) iso-cost).
Figure 18 is the headline cluster comparison. Against Baseline-A100 at iso-power, Splitwise-AA delivers $2.15\times$ throughput at the same cost; at iso-cost, Splitwise-AA gives $1.4\times$ more throughput than Baseline-H100 (at 25% more power, 2× space) — the "older cheaper GPUs can win on Perf/$" result.
6. 论证链 (Argument chain, paper-internal) #
| Step | Claim | Support within paper |
|---|---|---|
| 1 | LLM inference has two phases with opposite resource profiles | §II-B mechanics + Fig 5/6/7/8/9 characterization (Insights III–VI) |
| 2 | Hardware compute has outpaced memory, so a single GPU choice mis-serves the token phase | Table I ratios (Insight VII from Table IV) |
| 3 | Therefore split phases onto phase-matched machine pools with a two-level scheduler | §IV design: CLS/MLS, prompt/token/mixed pools |
| 4 | Splitting is only viable if the inter-machine KV-cache transfer is cheap | §IV-C layer-wise async overlap over InfiniBand/MSCCL++ |
| 5 | Measured transfer overhead is negligible ($0.8\%$ E2E) | §VI-A, Fig 14/15 |
| 6 | So provisioned clusters beat homogeneous baselines on throughput/cost/power | §VI-B/C, Fig 18/19 (1.4×/20% lower cost; 2.35×; HHcap −25% power) |
| 7 | And the design is robust to workload/model shifts via the elastic mixed pool | §VI-D, Fig 20 (≤7% setback on cross-workload) |
7. 实现 cross-reference #
The KV-cache transfer prototype is a public vLLM pull request built on MSCCL++; the cluster simulator is the open-source SplitwiseSim. Specific line-level anchors are not reproduced here — [实现未公开] at file:line granularity in the paper, but the artifact is archived at Zenodo DOI 10.5281/zenodo.11003049 (MIT code / CC-BY data), with the prototype on the vLLM repo [ref 1] and SplitwiseSim at [ref 20].
核心技术壁垒 (single hardest-to-replicate insight). The layer-wise asynchronous KV-cache transfer (§IV-C, Fig 11). Naively, one waits for the whole prompt phase to finish, then ships the entire KV-cache before the token phase can start — a serialized bubble that scales with prompt length and inflates second-token latency by 64%. The barrier is realizing the KV-cache for layer $l$ is finalized the moment layer $l$'s prompt compute completes, so an async one-sided put can fire per-layer and overlap with layer $l{+}1$'s compute. This requires: (a) a GPU-driven zero-copy communication path (MSCCL++ one-sided put over InfiniBand, no receiver-issued recv), (b) per-layer/per-request semaphore synchronization for correctness (each batched prompt request gets its own semaphore since it may route to a different token machine), and (c) a size-aware policy that reverts to serialized transfer for small prompts where the layer-wise sync interference would exceed the hidden latency. Reproducing the throughput numbers is easy; reproducing the near-zero perceived overhead depends entirely on getting this overlap and its synchronization right.
Paper's Figure 11, verbatim (caption: "Optimizing KV-cache transfer in Splitwise"; (a) serialized, (b) layer-wise async overlap).
Figure 11 contrasts the serialized Gantt (a) — transfer strictly after prompt, strictly before first token-phase step — against the layer-wise schedule (b) where each layer's transfer runs concurrently with the next layer's compute, collapsing the visible transfer to a small tail.
关键实现细节 (easy-to-miss tricks).
- Simultaneous pair assignment. The CLS assigns the prompt and token machine at the same scheduling instant, not sequentially — this is what makes the KV-cache transfer overlappable with prompt compute rather than a serial post-step.
- Block-contiguity coalescing. KV-caches ship block-by-block in vLLM, but Splitwise coalesces contiguous KV blocks that share the same semaphore to minimize the number of InfiniBand transfers.
- Minimum serving hardware. Both models run TP-8 on a single 8-GPU DGX (A100 or H100); Table I lists 80GB HBM per GPU. Token machines can be lower-tier (A100 for token pool in Splitwise-HA, or power-capped H100 in Splitwise-HHcap).
- KV-cache bytes/token → concurrency. From §4's estimate (~2.6–4.0 MB/token dense upper bound), the transferable per-request state and the token machine's memory capacity set max concurrency; Fig 7 shows token-phase memory growing with active batched tokens until OOM at batch 64.
- Prefill vs decode bottleneck. Prompt (prefill) is compute/power-bound and throughput-caps past 2048 batched tokens (Fig 6a); decode (token) is memory-bound and scales with batch (Fig 6b) — the entire motivation for disaggregation.
- Continuous / prefix caching friendliness. Token machines use pure continuous batching; the mixed pool uses mixed continuous batching. The paper explicitly does not reuse KV-cache across requests (§III) to emulate a security-isolated cloud service.
- At what prompt:output ratio does phase-splitting stop paying off? Coding (115× imbalance) makes token pools tiny; workloads with balanced ratios may see the mixed pool dominate and Splitwise "devolve into the iso-count baseline" (§VI-E).
- Can the phase-matched-hardware idea transfer beyond NVIDIA GPUs? The paper speculates AMD MI-250 or HBM-equipped Intel Sapphire-Rapids CPUs as token machines but leaves it to future work (§VII).
- Does a lower-bandwidth interconnect (RoCE/Ethernet, or InfiniBand-through-CPU for H100↔A100) preserve the near-zero transfer overhead? The authors argue a 10× slower link would still be beneficial given the overlap, and KV-cache compression could further cut bandwidth (§VII).
- Headline reconciliation gap. The abstract/§I advertise "1.4× at 20% lower cost" and "2.35×", while §IX conclusion states "1.76× better throughput with 15% lower power at same cost" — the headline pairs differ across sections and are not reconciled in the source.
§8 Serving deployment considerations #
§9 Open questions #
Appendix: 模型架构图 #
N/A — Splitwise is a serving/cluster deployment technique, not a model release with reference model code. The paper's own artifacts are a KV-cache-transfer patch to vLLM and the SplitwiseSim event-driven simulator; there is no modeling_*.py to extract per-block attention/FFN diagrams from. The relevant "architecture" is the cluster/scheduler system diagram embedded in §3 (Fig 10) and the pool state machine.