Understanding Bottlenecks for Efficiently Serving LLM Inference With KV Offloading — L2 #

§1 TL;DR #

推导 $\kappa_{\text{crit}} = \kappa_M \times \kappa_{HW}$ 分析框架，将 KV offloading 下 prefill 从 compute-bound 转 memory-bound 的临界点分解为模型因子和硬件因子；实证表明真实工作负载超过该阈值 1–2 个数量级，GPU 仅消耗 22–28% TDP，99% 延迟花在 PCIe 传输上。

§2 Q1 / Q2 / Q3 #

Q1 — 痛点 #

KV cache offloading 通过将 KV 存入 CPU DRAM 解锁长上下文推理，但 PCIe 带宽（实测仅 15 GB/s，峰值的 23%）远低于 HBM（TB/s 级），导致 prefill 从 compute-bound 退化为 memory-bound。现有调度器（vLLM iteration-level scheduling）和架构（prefill-decode disaggregation）的设计假设——prefill 是 compute-intensive——被彻底打破。MoE 模型因计算量减少但 KV 不减而更加脆弱。

Q2 — 方法 #

提出 $\kappa_{\text{crit}}$ 分析框架，将瓶颈临界点分解为两个独立因子：

$$\kappa_{\text{crit}} = \underbrace{\frac{F_{\text{pf}}}{B_{\text{kv}}}}_{\kappa_M \text{ (model)}} \times \underbrace{\frac{\text{BW}_{\text{PCIe}}}{C_{\text{eff}}}}_{\kappa_{HW} \text{ (hardware)}}$$

当工作负载的 $\kappa_{\text{ratio}} = K/T > \kappa_{\text{crit}}$ 时，prefill 进入 memory-bound 区域。框架将模型架构（GQA vs MLA, Dense vs MoE）和硬件规格（PCIe gen, GPU 算力）解耦，并进一步推导 VRAM-constrained 调度模型（Eq. 9–10）来量化 scheduler 效率损失。

核心技术壁垒: 将 roofline model 扩展到 KV offloading 场景的洞察——arithmetic intensity 不再由 model FLOPs/param 决定，而是由 $T/K$ 的 workload ratio 决定。这使得同一硬件上同一模型在不同 workload 下跨越 compute/memory bound 分界线，且该分界线随 GPU 迭代恶化（计算增速快于互联带宽）。

Q3 — 结果 #

• $\kappa_{\text{crit}}$ 实测值仅 1–2（远低于理论 7.8–14.3），因 PCIe 有效带宽仅峰值 23%
• ShareGPT median $\kappa_{\text{ratio}} = 100$，NarrativeQA = 5000，FinQA = 10000——全面超过阈值
• Qwen3-235B 在 $K=65$K, $T=64$ 时 $P_{OH}=86$（99% 时间在传输）
• GPU 平均功耗仅 TDP 的 22–28%
• NVLink C2C (900 GB/s) 将 $\kappa_{\text{crit}}$ 提升 5.3×，统一 HBM 提升 48×，但 document Q&A 仍超阈值

§3 架构 / 方法图 #

Paper's Figure 1, verbatim (caption: "KV cache offloading transfers computed representations from GPU VRAM to CPU DRAM. When serving the second request (green), only the novel token ('ocean') requires computation; cached tokens are loaded via PCIe.").

Figure 1 展示了 KV offloading 的核心数据流：第一次请求的 KV 被 offload 到 CPU DRAM，后续请求只需计算 novel tokens 的 KV 并从 CPU DRAM 加载已缓存部分。PCIe 成为瓶颈的根本原因——大量 cached tokens 需要跨越带宽瓶颈回到 GPU。

Paper's Figure 4, verbatim (caption: "Roofline models for Qwen3-235B-A22B (left) and Deepseek-V3 (right) on NVIDIA B200, H100 and A100 for current PCIe bandwidths. Vertical lines indicate κ_crit for specific hardware.").

Roofline 图直观展示了随 $\kappa_{\text{ratio}}$ 增大，操作点从 compute ceiling 滑入 bandwidth ceiling 的过程。DeepSeek-V3 的 MLA 将 compute-bound 区域延伸到 $\kappa_{\text{ratio}} \approx 40$，但两个模型在 $\kappa_{\text{ratio}} > 100$ 时均进入 bandwidth-limited 区域。

§4 作者证明 #

符号表 #

方程物理意义 #

Eq. 3 (TTFT decomposition): $\text{TTFT} = \frac{K \cdot B_{\text{kv}}}{\text{BW}_{\text{PCIe}}} + \frac{T \cdot F_{\text{pf}}}{C_{\text{eff}}}$

两项之和：PCIe 传输大量 cached KV 的时间 + GPU 计算少量 new tokens 的时间。假设无法完全 overlap（实测 $t_{\text{PCIe}} \gg t_{\text{GPU}}$ 时 overlap 收益可忽略）。

Eq. 6 ($\kappa_{\text{crit}}$ factorization): $\kappa_{\text{crit}} = \frac{F_{\text{pf}}}{B_{\text{kv}}} \cdot \frac{\text{BW}_{\text{PCIe}}}{C_{\text{eff}}}$

将 roofline 的 compute-bandwidth 交叉点（$AI = C/BW$）在 KV offloading 场景下重新表达为 $K/T$ 的临界值。乘法结构使模型优化（增大 $\kappa_M$）和硬件优化（增大 $\kappa_{HW}$）可独立推进。

Eq. 10 (Scheduled tokens): $T_{\text{sched}} = N_{\text{max}} \cdot T \approx \frac{V_{\text{eff}}}{\kappa_{\text{ratio}} \cdot B_{\text{kv}}}$

VRAM 容量（而非 token budget）决定每 iteration 实际可调度的 prefill tokens 数量，且与 $\kappa_{\text{ratio}}$ 成反比。这解释了为什么 NarrativeQA 仅调度 532 tokens（4K budget 的 13%）。

单调性 / 凸性分析 #

$\kappa_{\text{crit}}$ 关于 $\text{BW}_{\text{PCIe}}$ 单调递增，关于 $C_{\text{eff}}$ 单调递减——更快的 GPU 使问题恶化，更快的互联使问题缓解。$U$ 关于 $\kappa_{\text{ratio}}$ 单调递减（$U = 1/(1 + P_{OH})$, $P_{OH} \propto \kappa_{\text{ratio}}$），无内部极值点，边界行为为 $\kappa_{\text{ratio}} \to \infty$ 时 $U \to 0$。

6 Minimum Checks #

1. Dimensional consistency: $\kappa_{\text{crit}}$ is dimensionless — $[F_{\text{pf}}]/[B_{\text{kv}}]$ = FLOP/byte, $[\text{BW}]/[C]$ = byte/FLOP, product is pure ratio ✓
2. Boundary: $K=0$: no offloading → $\kappa_{\text{ratio}} = 0 < \kappa_{\text{crit}}$ → compute-bound, consistent with standard prefill ✓
3. Boundary: $T \to 0$: $\kappa_{\text{ratio}} \to \infty$, always memory-bound → pathological (single-token prefill of huge cached context) ✓
4. Case study LLaMA-405B on H100: $F_{\text{pf}} = 810$ GFLOP, $B_{\text{kv}} = 516$ KB, $\text{BW} = 64$ GB/s, $C = 2000$ TFLOP/s. $\kappa_M = 810G/(516K) = 1.57M$ FLOP/byte ≈ 1.42 (units: GFLOP/KB → 1.42 in normalized units). $\kappa_{HW} = 64G/2000T = 3.2 \times 10^{-5}$ byte/FLOP. Product ≈ 45–50 → matches paper's ~50 ✓
5. Measured bandwidth correction: using 15 GB/s instead of 64 GB/s → $\kappa_{\text{crit}}$ drops by 64/15 ≈ 4.3×, from ~50 to ~12, matches paper's "sustained ~12" ✓
6. Eq. 10 NarrativeQA: $V_{\text{eff}} = 92$ GB, $K=57000$, $T=12$, $B_{\text{kv}} = 328$ KB → $N_{\text{max}} = 92G/(57000 \times 328K) \approx 5$, $T_{\text{sched}} = 5 \times 12 = 60$... paper reports 532 for mixed workload (some requests have larger T) — directionally consistent with VRAM-limited regime ✓

§5 实验与数据 #

System scope #

• Stage coverage: Prefill only (decode excluded by design: output limited to 1 token)
• Serving mode: Continuous batching via vLLM v0.10.1 + LMCache v0.3.5
• Parallelism: TP (8× H100 SXM5), no EP/PP/SP analysis
• Deployment: Single-node, 8-GPU

Workload $\kappa_{\text{ratio}}$ distributions #



Paper's Figure 3, verbatim (caption: "Distributions of prefill tokens (T), reused KV tokens (K), and the ratio κ_ratio for ShareGPT (top), NarrativeQA (middle), and FinQA (bottom). Median κ_ratio: ShareGPT 100, NarrativeQA 5000, FinQA 10000").

三个数据集展示了逐级恶化的 $\kappa_{\text{ratio}}$：multi-turn conversation（ShareGPT）中 KV 增长缓慢，document QA 中整个文档被缓存而 query 极短，$\kappa_{\text{ratio}}$ 高达 10000。这些分布证明了 $\kappa_{\text{crit}}$ 被系统性超越。

PCIe overhead 实测 #



Paper's Figure 6, verbatim (caption: "PCIe overhead of KV Cache offloading under varying number of prefill tokens and KV cache size. Qwen with K=65k, T=64 exhibits P_OH=86, corresponding to 99% of execution time spent on PCIe transfers.").

热力图清晰展示了 $P_{OH}$ 随 $K$ 增大 / $T$ 减小而指数级增长。MoE 模型（Qwen3）在所有配置下 overhead 均高于 Dense 模型（Llama），验证了 §5.2 的分析预测。

Framework validation #



Paper's Figure 5, verbatim (caption: "Measurements of κ_crit.").

实测 $\kappa_{\text{crit}}$（Llama=2, Qwen=1）远低于峰值带宽假设下的理论值（14.3, 7.8），但使用实测 15 GB/s 带宽修正后（3.3, 1.8）与实测吻合。这验证了框架的正确性并突显了使用实测带宽的重要性。

Scheduler & power underutilization #

Metric	ShareGPT (70 RPS)	NarrativeQA (70 RPS)
$T_{\text{sched}}$ (tokens/iter)	4064	532
Budget utilization	~100%	~13%
Avg GPU power (W)	196	152
% of TDP	28%	22%

NarrativeQA 的极端 $\kappa_{\text{ratio}}$ 导致 VRAM 在 token budget 饱和前耗尽，scheduler 被迫闲置 87% 的预算。GPU 功耗仅达 TDP 的 22%，意味着数据中心电力/散热基础设施大量过剩。

硬件/模型优化效果 #



Paper's Figure 7, verbatim (caption: "Roofline models for Qwen3-235B-A22B (left) and DeepSeek-V3 (right) on NVIDIA GB200 with NVLink C2C and theoretical unified HBM. NVLink C2C increases bandwidth 14× versus PCIe 5.0, raising κ_crit by 5.3×. Unified HBM provides an additional 9× improvement.").

NVLink C2C 将 Qwen3 的 $\kappa_{\text{crit}}$ 从 3.1 提升到 41.5，但 document QA（$\kappa_{\text{ratio}}=5000$）仍超出 120×。只有统一 HBM（$\kappa_{\text{crit}}=370$ for Qwen, 1700 for DeepSeek-V3）才能让 MLA 模型的 document QA 接近 compute-bound。

Scheduling optimization direction #



Paper's Figure 8, verbatim (caption: "Utilization-aware scheduling example. FIFO scheduling (left) underutilizes the token budget due to VRAM constraints. Utilization-aware scheduling (right) reorders requests to saturate the token budget, improving GPU efficiency while introducing fairness concerns.").

FIFO 调度在 VRAM 约束下仅利用 3/5 token budget，而 utilization-aware 调度通过重排请求（优先调度低 $\kappa_{\text{ratio}}$ 请求）将利用率提升到 5/5，但引入公平性问题。

§5-ext Scheduling & Resource Management #

Scheduling decisions #

Aspect	Paper's approach
Granularity	Request-level (iteration-level scheduling, per-request token budget)
Preemption	Not explored; vLLM baseline supports preemption but paper focuses on admission
Admission control	VRAM-capacity-gated: requests deferred until KV fits in VRAM
Fairness	Identified as open problem; proposed aging credits + weighted fair queueing

Memory management #

Aspect	Current (vLLM + LMCache)	Problem
KV allocation unit	PagedAttention blocks (vLLM)	Budget accounts tokens, not bytes
Fragmentation	Paged → low internal fragmentation	VRAM exhausted by large $K$ before budget saturates
Eviction	LRU in LMCache (CPU DRAM)	CPU DRAM abundant; VRAM is the binding constraint
Swap	CPU DRAM → GPU VRAM on demand	PCIe bandwidth is the bottleneck

核心问题：scheduler 的 token budget 与 VRAM consumption 解耦——一个 $K=65$K, $T=32$ 的请求仅消耗 32 token budget 但需要 33 GB VRAM。

§5-ext Workload Characterization #

Workload regime	Offloading system	Baseline (no offloading)	Why
Short prompt, low $\kappa_{\text{ratio}}$ (<2)	Compute-bound, behaves like standard prefill	Same	$t_{\text{PCIe}} < t_{\text{GPU}}$
Multi-turn conversation (median $\kappa_{\text{ratio}}$=100)	Memory-bound; 88% time on PCIe	N/A (KV doesn't fit VRAM for all users)	$\kappa_{\text{ratio}} \gg \kappa_{\text{crit}}$
Document QA (median $\kappa_{\text{ratio}}$=5000)	Severely memory-bound; 99% time on PCIe, 13% budget utilization	Cannot serve (OOM)	Extreme $K/T$ mismatch
MoE + high $\kappa_{\text{ratio}}$	Worse than Dense (lower $\kappa_M$)	N/A	MoE reduces $F_{\text{pf}}$ faster than $B_{\text{kv}}$
MLA model (DeepSeek-V3)	Better; $\kappa_{\text{crit}}$ 4.6× higher than GQA MoE	N/A	$B_{\text{kv}}$ reduced 2.7× by low-rank compression

Paper wins: analytical prediction; loses: cannot demonstrate real MLA benefits (DeepSeek-V2 implementation issues) and does not propose a working scheduler implementation.

§6 论证链 #

Step	Claim	Evidence	Depends on
1	TTFT = $t_{\text{PCIe}} + t_{\text{GPU}}$ (sequential, overlap negligible when $t_{\text{PCIe}} \gg t_{\text{GPU}}$)	Eq. 3–4; measured overlap impact <1% for typical workloads	Physical constraint: GPU needs KV in VRAM before computing attention
2	$\kappa_{\text{crit}} = \kappa_M \times \kappa_{HW}$ factorizes transition point into model × hardware	Algebraic rearrangement of $t_{\text{PCIe}} > t_{\text{GPU}}$ (Eq. 5→6)	Step 1
3	Real $\kappa_{\text{crit}}$ is 1–2 (not 7–76) because sustained PCIe is 23% of peak	Fig. 5: measured $P_{OH}=1$ at $\kappa_{\text{ratio}} \approx 1$–2; recalculated with 15 GB/s matches	Step 2 + empirical bandwidth measurement
4	Real workloads' $\kappa_{\text{ratio}}$ (100–10000) exceed $\kappa_{\text{crit}}$ by 50–5000×	Fig. 3: ShareGPT/NarrativeQA/FinQA CDFs	Step 3 + dataset analysis
5	Scheduler under-utilizes token budget because VRAM (not budget) is binding	Table 4: NarrativeQA schedules 532/4000 tokens; Eq. 10 predicts correctly	Step 4 + Eq. 9–10
6	GPU power drops to 22–28% TDP → infrastructure overprovisioned	Table 5: 152–196W measured vs ~700W TDP	Step 5 (low utilization → low power)
7	NVLink C2C raises $\kappa_{\text{crit}}$ 5.3×, unified HBM 48×, but only HBM + MLA approaches sufficiency	Fig. 7: roofline extension shows extended compute-bound region	Step 2 (scale $\kappa_{HW}$ with new bandwidth)

§7 实现 cross-reference #

Baseline implementation: vLLM v0.10.1 + LMCache v0.3.5 (open-source).

• vLLM iteration-level scheduler with token budget: vllm/core/scheduler.py (v0.10.1) — budget-based admission, VRAM tracking
• LMCache KV offloading integration: LMCache/lmcache (v0.3.5) — CPU DRAM KV store with PCIe transfer

[实现未公开] — The paper proposes utilization-aware scheduling and $\kappa$-based routing as design implications but does not release an implementation. The $\kappa_{\text{crit}}$ framework itself is purely analytical (no code artifact).

核心技术壁垒 (§7 dedicated paragraph): 本文的核心 insight 是将 KV offloading 场景下的 prefill 性能问题重新框定为 roofline 问题，其中 arithmetic intensity 不由模型本身决定，而由 workload 的 $K/T$ 比例动态决定。这意味着：(1) 同一模型同一硬件的性能可因 workload 变化差异 100×；(2) GPU 越快问题越严重（$\kappa_{HW}$ 随 $C_{\text{eff}}$ 增大而减小）；(3) scheduler 的 token budget 会被 VRAM 约束架空。这个洞察不需要复杂实现，但需要对 serving system 设计理念的根本性转变。

关键实现细节:

1. Sustained PCIe bandwidth = 23% of peak: 任何基于 $\kappa_{\text{crit}}$ 的调度决策必须使用实测带宽（15 GB/s on H100），而非 spec sheet 的 64 GB/s。NUMA effects、CPU memory copy overhead、transfer granularity 是三个主因。
2. VRAM-budget decoupling: 当前 vLLM scheduler 将 offloaded requests 的 prefill token 数计入 budget 但不追踪其 VRAM demand（$(K+T) \cdot B_{\text{kv}}$），导致 VRAM OOM 先于 budget exhaustion。修复需要在 admission control 中引入 VRAM-aware accounting。

§8 API & Usability #

• User-facing API: 标准 vLLM OpenAI-compatible API（无额外配置暴露给用户）
• Config surface: LMCache 配置 offloading 策略（CPU DRAM path），vLLM 配置 token budget size
• Migration cost: 已使用 vLLM 的部署只需添加 LMCache 集成（已有上游支持）

Paper 不提供新的 serving system，而是分析现有系统（vLLM + LMCache）的瓶颈并提出设计方向。

§9 Adoption & Ecosystem #

• Upstream status: 分析框架本身未合并到任何 serving system；LMCache 已是 vLLM 生态的组件
• Production deployment: 未声明
• Downstream forcing: 提出的 utilization-aware scheduling 需修改 vLLM scheduler 的 admission control 逻辑；硬件建议（NVLink C2C, unified HBM）指向 NVIDIA Grace Blackwell/Grace Hopper 平台

§10 Deployment Context (实践上下文) #

• Serving stage: Prefill only（paper explicitly scopes to prefill; decode 不受此分析影响因为 decode 已是 memory-bound）
• Concurrency regime: Low concurrency (1–5 requests) due to VRAM exhaustion; high $\kappa_{\text{ratio}}$ workloads cannot co-schedule more than 1–2 concurrent requests on 80 GB GPU
• Hardware affinity: A100 (PCIe 4) 反而比 B200 (PCIe 5) 更不容易进入 memory-bound（$\kappa_{HW}$ 更高），但绝对性能更低。Grace Hopper/Blackwell (NVLink C2C) 获益最大。
• Ecosystem integration: 分析适用于任何使用 KV offloading 的 serving system（vLLM, SGLang, TRT-LLM + LMCache/CacheGen）
• Migration path: 无代码变更需要；paper 提供的是分析工具和设计原则，帮助判断是否值得部署 offloading、选择什么硬件

§12 Software → Hardware Reverse Implication #

Triggered: paper 直接讨论了 hardware-proximal 优化（interconnect bandwidth, unified HBM architecture, chiplet topology implications）。

Paper argues for:

1. Unified HBM: 消除 CPU-GPU 互联瓶颈，将 KV cache 置于共享 HBM 中。需要解决热管理、die area、coherency 协议挑战。$\kappa_{\text{crit}}$ 可达 370–1700，使多数 workload 回到 compute-bound。
2. NVLink C2C 泛化: 当前仅 Grace Hopper/Blackwell 支持 900 GB/s C2C；paper implicitly argues 所有 inference GPU 都应提供高带宽 CPU-GPU 互联（而非仅依赖 PCIe）。
3. ISA/cache implication: 更快的 GPU（B200+）在 KV offloading 场景下反而更受限，暗示未来 inference-optimized GPU 可能需要牺牲 peak FLOP/s 来换取更平衡的 bandwidth/compute ratio，或者在芯片内部集成更大容量的低延迟 KV buffer。