Note on scope: the task-provided related set differs from the L2's declaredrelated(which pins vLLM/SGLang/FlexGen). Two provided IDs are alias-normalized:2310.0724→ 2310.07240 (CacheGen),2401.0967→ 2401.09670 (DistServe).2305.05920(FastServe) has an L2 in scope and is cited below. All eight peers arecategory: framework.2312.07104(SGLang) L2 was in the allowlist but its hash field is literalin-file, recorded asMISSINGin frontmatter; its content is fully read and cited.
The eight peers split into three concentric rings around CachedAttention, whose
thesis is: *reuse KV across multi-turn turns by storing it on cheaper tiers, and
make the tiering pay off via the scheduler*.
Ring 1 — KV-reuse / prefix-reuse serving (the direct siblings).
radix tree with LRU eviction and cache-aware scheduling. It is the closest
mechanism peer: both convert "prefill history" into "load cached KV," and both
serve multi-turn chat. The delta is tier (HBM-only tree vs DRAM+SSD) and
eviction oracle (LRU-history vs scheduler-future).
Variables at the cluster manager, with dynamic (runtime-generated) prefix
detection — a superset reuse target that includes but is not limited to chat
history.
Ring 2 — tiered / offloaded KV placement + transport (the cost-tier peers).
slow remote tier" problem, but its slow tier is the network (single-digit Gbps
cloud links), and its lever is compression (KV-as-video, 3.5–4.3× size cut),
not scheduler-aware placement. It is CachedAttention's orthogonal complement.
[2403.11421] both spill KV off HBM (FastServe: proactive
swap to host memory driven by an ENST oracle; FastDecode: KV permanently resident
on remote CPU). FastServe's ENST-driven proactive swap is the *closest conceptual
cousin* to CachedAttention's scheduler-aware fetch/evict — both mine the scheduler
for a future-access oracle.
Ring 3 — engine substrate + orthogonal serving axes (the context peers).
memory substrate CachedAttention explicitly borrows ("block-based storage similar
to vLLM [21]", L2:2403.19708#7-实现-cross-reference); it is the shared foundation,
not a competitor.
a fully orthogonal axis (phase placement, not KV lifetime). It matters because
CachedAttention's win is a prefill-side win, and DistServe is the canonical
prefill/decode-decoupling design its gains could stack on.
placement framework — same category: framework, but different stage (training
vs serving). It is included only as a category sibling; its overlap with
CachedAttention is the shared "disaggregate/place by role" design instinct, not
a KV-reuse mechanism.
What's genuinely new in CachedAttention.
eviction from history; CachedAttention reads the serving job queue to know
future accesses and drives both prefetch and eviction from it, landing
99.6% of hits in DRAM vs ~0.5% for LRU/FIFO [2403.19708]. SGLang
uses the opposite discipline — LRU leaf-first eviction with reference
counters [2312.07104]. This is the sharpest single delta in
the cluster: SGLang's cache is history-driven and HBM-bound; CachedAttention's
is future-driven and disk-bound.
Parrot reuse in HBM only (SGLang's tree lives in one GPU pool
[2312.07104]; Parrot forks context inside the engine's paged
blocks [2405.19888]). CachedAttention's storage-tier ablation
shows HBM-only ≈ 0% hit at scale and only SSD lifts hit to 71–90%
[2403.19708] — i.e. it directly refutes an HBM-only reuse
design for long-horizon multi-turn workloads.
of cached KV. This is a net-new correctness contribution (PPL within 0.02 of
token-truncation; naive coupling explodes to >10³) [2403.19708].
Incremental / borrowed.
[2309.06180], acknowledged in
same overlap principle FastServe uses for proactive swap
[2305.05920] and FastDecode for its S/R pipeline
[2403.11421] — CachedAttention's contribution is the
layer-wise granularity and the buffer-sizing formula, not the overlap idea.
Contradictory / tension.
scheduler-aware (future) eviction is necessary to keep hits off the slow tier
[2403.19708], while SGLang ships LRU and proves its *greedy
longest-prefix schedule* reaches 96% of optimal hit rate
矛盾根源 — 不同 tier 假设: SGLang's LRU operates entirely in HBM where every
hit is equally fast, so hit rate is the only objective; CachedAttention's hits
span DRAM (fast) vs SSD (10× slower), so where a hit lands matters as much as
whether it hits. LRU that is near-optimal on a single-tier cache is provably
bad on a two-tier cache — both are correct within their own tier model.
Attack 1 — the "scheduler oracle" is only as good as output-length predictability.
CachedAttention's prefetch/eviction windows ($L_{pw}, L_{ew}$) assume the job queue
tells you when a session's KV is next needed [2403.19708]. But
in multi-turn chat the inter-turn think time is user-controlled and unbounded — a
session can idle for minutes, then reappear. The look-ahead window is over *queued
jobs, not future arrivals*; a session with no pending job is invisible to the
oracle. SGLang concedes exactly this class of failure (its footnote admits
unpredictable output tokens and starvation are unsolved
[2312.07104]). CachedAttention's CCpUT = DSpUT · CCpS
[2403.19708] sizes capacity for the TTL window but says nothing
about the tail of idle sessions that overflow it — the >99.6% DRAM-hit number is a
steady-state-load figure that the arrival-rate sweep (hit 82→77% at 0.5→2.0/s
[2403.19708]) already shows degrading.
Attack 2 — no head-to-head against an HBM-tree baseline of the same era.
The storage-tier ablation compares against "HBM-only" as a strawman (≈0% hit)
[2403.19708], but a fair rival is SGLang/RadixAttention, which
also keeps only recent KV in HBM yet reaches 52–74% production hit on multi-turn
chat [2312.07104]. CachedAttention never runs against a tree-cache
baseline — so the claim that tiering (not a better HBM policy) is the binding win is
under-evidenced against the strongest same-tier competitor.
Attack 3 — the RoPE-decoupling trick is silently model-class-bound.
It "only works with relative/rotary PE, not absolute PE"
[2403.19708]. The lossless-truncation headline (PPL
<0.02) therefore does not generalize to absolute-PE models; the paper presents it as
a general truncation solution but it is an RPE-only patch.
Attack 4 — PCIe/SSD bandwidth sensitivity is measured, not modeled.
The L2 itself flags that there is no closed-form throughput model, so the reported
6.8–7.8× is not portable to a different tier hierarchy
[2403.19708]. On a cluster where reused KV lives on the network
rather than local SSD, CacheGen's own data shows transfer can rival full recompute
[2310.07240] — a regime
CachedAttention (single-node, local SSD) never tests.
Paradigm position. CachedAttention sits at the intersection of two 2023–24
paradigm shifts in serving: (a) *KV reuse is the dominant lever for
prefix/history-heavy workloads* (SGLang [2312.07104],
Parrot [2405.19888] both make the "72–99% of tokens are repeated"
argument), and (b) KV must escape HBM onto a memory/storage hierarchy
(FastServe host-memory swap [2305.05920], FastDecode remote
CPU [2403.11421], CacheGen remote store
CachedAttention is the paper that fuses the two: reuse target = multi-turn history,
storage tier = DRAM+SSD, and it is the one that makes the fusion economical via the
scheduler-cache coupling. That coupling is its unique niche — no other peer converts
the serving scheduler into a cache oracle across a multi-tier store.
Adoption evidence. Weak. CachedAttention is [实现未公开] — no public repo
[2403.19708]. Contrast the peers: vLLM is the de-facto
industry engine [2309.06180], SGLang's RadixAttention
was upstreamed and is public [2312.07104], CacheGen
[2310.07240] and DistServe
[2401.09670] are both open-source. The scheduler-aware
tiering idea has since diffused into production stacks (LMCache, KV-offloading in
vLLM/SGLang) but CachedAttention itself, being closed and single-node, is cited as a
design reference rather than a deployed system — the same fate as FlexRLHF, whose
un-open-sourced ideas were absorbed by OpenRLHF
reused KV can be compressed 3.5–4.3× without quality loss and stacks on top of
any float KV [2310.07240]; CachedAttention moves KV between DRAM
and SSD in full precision. Compressing the SSD tier would shrink $S_{kv}$,
directly widening both look-ahead windows
($L_{pw}=C_{mem}/S_{kv}$ [2403.19708]) — more sessions
protected per byte. Nobody combines a Belady-scheduler oracle with a compressed
cold tier.
layer).** CachedAttention is single-node local-SSD; CacheGen is the network-tier
specialist but uses reactive per-chunk bandwidth adaptation, not a scheduler
oracle [2310.07240]. A future-aware prefetch that pulls a
session's KV from a remote store before its next turn is queued — driven by
CachedAttention's window over a cluster scheduler — is unexplored.
CachedAttention's win is prefill-side (turn history reused → shorter prefill),
and DistServe disaggregates prefill onto its own pool with NVLINK-affinity KV
transfer [2401.09670]. A prefill pool that also owns the
multi-turn AttentionStore, so reactivated sessions skip both recompute and phase
interference, is a natural hybrid neither paper builds.
already computes a per-job "next scheduled time" for swap ordering
[2305.05920]; CachedAttention's window is a coarser
session-granular version. Merging ENST's finer per-job timing into
CachedAttention's tier-placement decision would give a quantitative eviction
priority instead of the current binary "inside/outside the window."
hit-rate erosion as DSpUT rises [2403.19708]; an adaptive
controller that switches between scheduler-oracle eviction (steady load) and a
recency/frequency hybrid (bursty, low-predictability load) is an unexplored knob
the fixed-window design cannot express.