LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference — L2 #

§1 TL;DR #

LMCache is an out-of-GPU KV cache layer that extracts KV cache from vLLM/SGLang and stores/shares it across a storage hierarchy (CPU, disk, Redis, remote, network) for two use cases: cross-query prefix reuse and prefill–decode disaggregation. Its core trick is moving KV at a large chunk granularity (default 256 tokens) instead of the engine's 16-token page, plus layer-wise compute–I/O overlap, yielding up to 15× throughput and ≥2× lower latency vs vLLM/commercial baselines.

§2 Q1 / Q2 / Q3 #

Q1 — 痛点 (Pain point) #

Production KV cache has outgrown GPU memory and is increasingly reused across queries, so it must live outside the GPU. But moving it out is hard for three interlocking reasons: (1) paged-attention engines store KV in 16–64 KB non-contiguous pages, and naive per-page transfer badly underutilizes bandwidth (a transfer must reach ~16 MB to saturate a 400 Gbps NIC; sub-1 GB/s with torch.save); (2) inference engines evolve too fast (one prominent LLM every ~4 days in 2025) — every engine update can change the in-GPU KV layout and break any caching library glued to it; (3) there is no management API for higher-level components (routers, ops teams) to locate/pin/evict/compress caches in a KV-aware way.

Q2 — 方法 (Method) #

A KV caching layer sitting between the engine and heterogeneous storage, with three matching design pillars:

1. Performance — batched chunk-level I/O (contiguous streaming GPU buffer + custom CUDA scatter/gather kernels), layer-wise compute–I/O pipelining on separate CUDA streams, asynchronous prefetch during queue wait, and zero-copy multi-destination writes via reference counting.
2. Standardized connector API — a 7-function interface split across vLLM's scheduler (3 fns) and model runner (4 fns), co-designed and co-maintained with the vLLM team, so LMCache tracks engine evolution without ad-hoc patches.
3. Controller API — a centralized manager + per-instance workers exposing lookup / move / clear / pin / compress (external) and batched_admit / batched_evict / batched_p2p_lookup (internal) for cache-aware routing, migration, and P2P sharing.

核心技术壁垒: the chunk-vs-page transfer redesign — grouping scattered pages across multiple layers into a contiguous streaming buffer with a custom CUDA kernel, then DMA-ing at chunk granularity, while keeping only a single-layer-sized GPU buffer alive via layer-wise pipelining. This is what turns 88 Gbps (vLLM native, per-page) into 400 Gbps (Table 5), and it is co-tied to the connector's layerwise start_load_kv / wait_load_kv hooks — hard to replicate without owning both the data-movement kernels and the engine-side hook contract.

Q3 — 结果 (Results) #

Up to 15× throughput over baselines. CPU-offload scenario: 1.9–8.1× smaller TTFT, 2.3–14× higher throughput at equal TTFT, 7–92% lower ITL (§8.2). Real-trace: TTFT down 3.7–6.8×, ITL down 19–58% at high QPS (§8.3). Central remote storage: 1.3–3× throughput (§8.4). PD disaggregation: mean TTFT 1.53–1.84× lower, mean ITL 1.12–1.66× lower (§8.5). Component isolation: CPU-load bandwidth 400 vs 88 Gbps (Table 5); async overlap gives 1.46× end-to-end reduction (§8.6). Portable to SGLang (§8.8).

---

§3 架构 / 方法图 #

LMCache positions itself as a distinct layer between engines and storage/network backends.



Paper's Figure 5, verbatim (caption: "LMCACHE sits between LLM inference engines and heterogeneous storage/network devices."). The three-tier stack — engine on top (vLLM/SGLang), LMCache in the middle owning the KV cache abstraction, and swappable storage backends (Mooncake, Redis, InfiniStore, …) at the bottom — is the whole thesis: KV cache becomes a first-class, engine-independent data structure with its own movement/management substrate.

The end-to-end component graph shows where scheduling and memory management actually live:



Paper's Figure 6, verbatim (caption: "End-to-end system workflow for LMCACHE."). Note the separation the framework enforces: the KV Connector is the only touch point with the engine's scheduler + model runner; inside a LMCache instance the Token Processor (decides how many tokens are new / prefix-matched), Event Manager (tracks query IDs, launches async layer-wise load events), and Storage Manager (looks up backend addresses, drives transfers) are distinct; the Transfer Channel + Memory Allocator feed four backends (PD / P2P / Remote / Local) over NVLink / RDMA / TCP; and the Cache Controller + LMCache Worker form a separate control plane used by the Router. Scheduling decisions (batching) remain in the vLLM scheduler — LMCache only feeds it extra "matched tokens" so cache hits change how many tokens need prefill.

The two supported deployment modes — cross-query offloading vs cross-engine transfer — are one abstraction with two data paths:



Paper's Figure 2, verbatim (caption: "LMCACHE supports both context caching (KV cache offloading and sharing across queries) and PD disaggregation (cross-engine transfer of KV caches"). (a) One engine reuses KV across requests via CPU/disk offload; (b) prefiller engines hand KV to decoder engines cross-engine. The reader should notice the same connector/storage-manager machinery serves both — the difference is only the backend chosen (Local vs PD/P2P).

Request lifecycle through the connector hooks (the scheduler→runner path the framework owns):

sequenceDiagram participant Q as Query participant S as vLLM Scheduler participant C as KV Connector participant R as Model Runner participant B as LMCache Backend Q->>S: arrives S->>C: get_num_new_matched_tokens(query) C->>B: lookup matched prefix tokens C-->>S: matched_tokens (or None → requeue, overlap I/O) S->>C: update_state_after_alloc(query, blocks) C->>C: build_connector_meta(scheduler_output) Note over R,B: model runner phase (layerwise) R->>C: start_load_kv(layer 0) loop each layer l R->>C: wait_load_kv(l) // sync layer l, prefetch l+1 R->>R: compute layer l R->>C: wait_store_kv(l-1); start_store_kv(l) end C->>B: chunked store of new KV

Scheduler discipline: the framework delegates batching (FCFS-style continuous batching) to vLLM; its only scheduling lever is returning None from get_num_new_matched_tokens to push a request back to the waiting queue so its I/O overlaps others' compute. Memory-manager allocation unit: the chunk (default 256 tokens = 16 pages of 16 tokens each), materialized in a contiguous streaming GPU buffer; the persistent buffer footprint is only one layer's KV cache (layer-wise pipelining).

---

§4 作者证明 #

无形式化作者证明 — 仅实证. This is a systems/experience paper with zero numbered display equations and no analytical throughput/latency model. There is no notation table (λ, μ, B, N_p) to reproduce; all claims are empirical (§8) or field-observational (§9).

What a formal model would have clarified (and is currently left to measurement):

1. The crossover point between "load KV from remote" and "recompute prefill" — §8.7/Fig 15 shows it empirically as a function of context length and bandwidth (256K tokens at 32 Gbps), but there is no closed form $L^* = f(\text{BW}, \text{model FLOPs})$. A model of the form "load wins when $\frac{S_{kv}}{\text{BW}} < T_{\text{prefill}}(L)$" would predict the crossover instead of sweeping.
2. The chunk-size sweet spot: default 256 tokens is asserted, not derived. Table 1's message-size→throughput curve is the empirical basis, but no optimization of chunk size vs per-transfer CUDA-launch overhead is given.
3. The dynamic-offloading duplication window trade-off (§5.3) is described qualitatively (smaller window ⇒ fewer duplicated pages but more allocation stalls) with no stall-probability model as a function of window size and arrival rate.

Because no equations exist, the standard 6 formal checks (notation / physical meaning / monotonicity / convexity / boundary optimum / first-order plug-in) are not applicable; the paper's rigor lives in ablations (Table 5, Fig 13/14) rather than derivations.

---

§5 实验与数据 #

The headline CPU-offload result across five models:



Paper's Figure 8, verbatim (caption: "Compared to basic vLLM, basic vLLM CPU offloading, and two commercial alternatives, LMCACHE has 1.9–8.1× smaller TTFT, and supports 2.3–14× higher inference throughput..."). Both TTFT and ITL rows show LMCache's curve staying flat while baselines blow up as QPS rises — the flatness is the point: larger CPU cache capacity ⇒ higher hit ratio ⇒ prefill work avoided. Notice Basic vLLM CPU Offloading disappears on Qwen3-Coder-480B (fails to run) and both commercial options can't host it at all — a completeness gap the baselines are hiding.

Component isolation proves the chunk-vs-page mechanism is the real cause, not incidental tuning:



Paper's Table 5, verbatim (caption: "LMCACHE achieves much higher loading bandwidth when loading KV cache from CPU memory, compared to vLLM's native CPU offloading."). The ~4.5× bandwidth gap (400 vs 88 Gbps) comes purely from transfer granularity: per-page copies pay CUDA-launch + metadata + completion-signal overhead on every 16-token page, whereas chunked copies amortize it. This is the single most load-bearing ablation in the paper.

Async compute–I/O overlap, visualized as a timeline:



Paper's Figure 13, verbatim (caption: "With request asynchronization, LMCACHE overlaps KV cache loading and inference computation (either prefill or decode)."). Top timeline (LMCache async) packs loading bars underneath compute bars; bottom (vLLM sync) serializes them. The measured effect is a 1.46× end-to-end reduction — the mechanism behind the flat ITL curves in Fig 8.

The regime where loading loses to prefill (the honest failure boundary):



Paper's Figure 15, verbatim (caption: "At network bandwidth of 32Gbps, LMCACHE's KV cache offloading only outperforms basic vLLM's prefill when input length is more than 256K tokens..."). At 32 Gbps, prefill is faster below 256K tokens — LMCache loses there and must be adaptive; at 64/128 Gbps loading wins everywhere. This directly motivates the "adaptive load-vs-compute" decision the paper flags as future work.

PD disaggregation tail-latency win:



Paper's Figure 12, verbatim (caption: "Compared to vLLM's native PD disaggregation, LMCACHE's PD disaggregation has significantly lower tail latency, and achieving 1.5–1.8× lower mean TTFT..."). The CDFs (top row zoomed to 0.96–1.00) show LMCache's tail pulled left. Root cause per §8.5: vLLM's native path uses NIXL page-by-page copy of scattered prefiller pages (bandwidth underutilized), while LMCache stages each chunk into a contiguous GPU buffer before transfer — again the chunk-vs-page theme.

---

§6 论证链 #

Step	Claim (paper-internal)	Support
1	KV cache now exceeds GPU memory and is reused across queries at growing rates	Real usage telemetry: Fig 1 (users ↑23×, stored size ↑5.6×), Fig 3 (non-GPU KV portion grows), Fig 4 (>19% users reuse tokens >1.5×)
2	Therefore KV must move out of GPU, but paged (16–64 KB) memory makes per-page transfer bandwidth-starved	§3.1.1 + Table 1: must reach ~16 MB to saturate 400 Gbps; sub-1 GB/s with torch serialization
3	So move KV at large chunk granularity with contiguous staging + custom CUDA kernels	§5.1 design; validated by Table 5 (400 vs 88 Gbps) and Fig 14 (PD transmit time 3.68 vs 4.47 s)
4	Overlap the (now-efficient) transfer with compute via per-layer CUDA-stream pipelining + async prefetch	§5.2 design; validated by Fig 13 (1.46× end-to-end reduction)
5	Decouple from fast-evolving engines via a co-maintained 7-fn scheduler/runner connector so the above survives engine churn	§6 design + Table 2; validated by 6-month upstream adoption (Dynamo, llm-d, AIBrix, production-stack)
6	Expose a control-plane API so routers/ops can act cache-aware (locate/move/pin/compress)	§7 design + Table 3; enables cache-aware routing / migration / P2P
7	End-to-end this yields up to 15× throughput and ≥2× latency across offload / remote / PD scenarios	§8.2–§8.8 across 6 models and 3 scenarios

---

§7 实现 cross-reference #

Source is open at https://github.com/LMCache/LMCache but no file:line anchors are given in the paper; treat concrete internals as [实现未公开] at the citation level. The connector API is upstreamed into vLLM (co-maintained), so the 7 functions in Table 2 are the reproducible contract:



Paper's Table 2, verbatim (caption: "Functions in LMCACHE's connector."). Scheduler side: get_num_new_matched_tokens (returns matched-token count, or None to requeue for I/O overlap), update_state_after_alloc, build_connector_meta. Model-runner side: start_load_kv / wait_load_kv / start_store_kv / wait_store_kv — the layerwise pairing is exactly what the §3 mermaid sequence encodes.

核心技术壁垒 (detail). The hardest-to-replicate insight is not any single function but the co-design of two things that must agree: (i) the chunk-staging data path — a custom CUDA kernel gathers scattered paged KV from multiple layers into one contiguous streaming buffer, DMA-offloads at chunk granularity, and on load DMA-fetches then scatters back into pages; and (ii) the layerwise hook contract where wait_load_kv(l) synchronizes layer l while kicking off layer l+1's prefetch, so only a single-layer-sized GPU buffer is ever pinned. Owning only one half (fast kernels or engine hooks) reproduces neither the 400 Gbps bandwidth nor the fixed-buffer memory bound. Anyone lacking the upstreamed vLLM hooks must fork the scheduler/runner — the exact ad-hoc maintenance burden §3.1.2 warns against.

关键实现细节 (easy-to-miss tricks).

1. get_num_new_matched_tokens returning None is not an error path — it is a deliberate admission-control lever that pushes a cache-hit request back to the waiting queue precisely so its (slow) KV load overlaps other requests' compute. Miss this and you serialize I/O.
2. Zero-copy via reference counting (§5.3): a multi-destination write (CPU→disk + CPU→remote simultaneously) increments a single refcount on the shared source instead of duplicating; the buffer frees at count 0 (the paper's PCB-counter analogy). Combined with dynamic offloading's three-pointer (start/current/end) window, which duplicates only a subset of free GPU pages to cap duplication ratio at the cost of possible allocation stalls.

---

§8 部署上下文 (Deployment context) #

• Serving stage: both prefill (prefix reuse lowers TTFT) and decode (delayed chunk-based storing of generated KV), plus a cross-engine orchestration mode (PD). It is a caching/movement layer, not a scheduler — batching stays in the host engine.
• Concurrency regime: shines at mid-to-high QPS where GPU-only prefix cache thrashes and CPU/remote capacity buys hit-ratio (Fig 8 gains widen with QPS). At low QPS / short contexts on slow links it can lose to plain prefill (Fig 15, 32 Gbps < 256K tokens).
• Hardware affinity: benefits scale with the transfer link — full-duplex PCIe / NVLink / RDMA let chunked parallel store+load run concurrently; low-bandwidth remote (≤32 Gbps) narrows the win. Supports NVIDIA/AMD/Ascend/TPU processors and NVLink/RDMA/TCP transports.
• Ecosystem integration: plugs into vLLM (primary, connector co-maintained upstream) and SGLang (§8.8, comparable to native offload but adds hierarchical storage); embedded in Dynamo, llm-d, AIBrix, vLLM production stack, KServe. Integration cost is a connector implementation, not an engine fork.
• Migration path: a shop on vLLM adopts LMCache by enabling the KV connector and pointing it at a backend (CPU / disk / Redis / Mooncake / S3 / InfiniStore / NIXL) — config-level, no custom attention ops, typically via the official Docker image (§9: most users never read the source).

Workload regime table (where it wins / loses):

Workload regime	LMCache	Baseline (vLLM)	Why
short prompts, low concurrency, slow remote link	can lose	competitive	prefill faster than remote load below crossover (Fig 15, 32 Gbps)
long prompts, high concurrency	large win (up to 15×)	thrashes GPU cache	CPU/remote capacity ⇒ high hit ratio; chunked load saturates BW (Fig 8)
mixed prefill–decode (PD disagg)	1.5–1.8× lower mean TTFT	page-by-page NIXL copy underutilizes BW	contiguous chunk staging before cross-engine transfer (Fig 12/14)

Software→hardware reverse implication: LMCache is hardware-proximal (KV physical layout control, DMA chunk staging, interconnect-aware parallel store/load). Its complexity argues for hardware/ISA primitives that make scattered-gather DMA of paged tensors cheap — e.g. a gather-DMA engine that reads a page-index list into a contiguous descriptor without a custom CUDA kernel, or cache/interconnect support to transfer non-contiguous KV pages at large-message efficiency. That would eliminate the streaming-buffer copy that the whole §5.1 chunk design exists to work around.