ForkKV: Scaling Multi-LoRA Agent Serving via Copy-on-Write Disaggregated KV Cache — L2 #

1. TL;DR #

Multi-LoRA agent serving breaks prefix caching: distinct adapters make identical-text KV caches diverge, so each agent keeps a redundant full cache and GPU memory saturates linearly. ForkKV physically splits KV into a shared base cache ($xW$) + tiny per-agent residual ($xA_i$), forks it with OS-style copy-on-write via a DualRadixTree, and reconstructs it in SRAM with a fused ResidualAttention kernel — up to 3.0× throughput at ~0.71% quality loss.

2. Q1 / Q2 / Q3 #

Q1 — 痛点 (pain). Modern agentic workflows (coding assistants doing ReAct-style reason→act loops, or MapReduce fan-out) run many specialized agents that share a massive static prefix (system prompt, codebase). LoRA lets these agents co-host on one base model cheaply (a rank-16 adapter on Llama3.1-70B is ~0.28% of the model, ~400MB vs 140GB). But each adapter's unique activations make the KV cache diverge across agents even for identical text, so classic prefix caching cannot share it. The system keeps an independent full KV cache per agent; memory scales linearly with agent count and saturates GPU capacity, collapsing batch parallelism. Concretely throughput for ReAct/MapReduce drops 90.8% / 90.1% as workflows scale 1→8.

Q2 — 方法 (method). Exploit LoRA's additive algebra $Y=xW+xA_iB_i$. Decouple the cache into a bCache ($xW$, shared globally, RoPE already applied) and a rCache ($xA_i$, per-agent, tiny because $r\ll n$). Since $r\ll n$, bCache is ~dozens× larger than rCache, so sharing it once amortizes memory across all agents. Three pieces make this work: (1) a DualRadixTree with a base tree (keyed by token ids) and a residual tree (keyed by token ids + agent id); (2) CoW fork semantics — a new agent prefix-matches to inherit the read-only bCache and copy-on-write-allocates only its rCache; (3) ResidualAttention, a fused Triton kernel that reconstructs the disaggregated cache inside SRAM.

核心技术壁垒 (the single hardest-to-replicate insight): the ResidualAttention kernel that reconstructs the full Key in on-chip SRAM using a deferred RoPE (RoPE cannot be applied to rCache because its dimension is $r\neq n$, so it is delayed until after $K_{res}B_k$ up-projection inside the attention loop) and then fuses the Value path via matrix associativity — pushing the up-projection $B_v$ out of the inner loop so it runs once at the end. This is what makes disaggregation computationally free instead of memory-negating; it is a non-obvious coupling of positional encoding, low-rank projection, and online-softmax accumulation that a reimplementer would most likely get wrong. See §7.

Q3 — 结果 (results). Built on SGLang v0.5.6 (~3K LoC + custom Triton kernels). Throughput 1.25–3.04× on ReAct and 1.68–2.60× on MapReduce across Llama3-8B / Qwen2.5-7B / Qwen2.5-14B on LooGLE/NarrativeQA/APIGen, with avg quality drop of only 0.71 F1 points (max 1.60). Root causes: 12.7× lower per-agent memory, 6.93× higher cache hit rate, 12.0× larger decode batch size. Full-reuse baseline (naively sharing across adapters) instead loses 5.40 avg / 21.95 on APIGen — showing the residual path is what preserves quality.

3. 架构 / 方法图 #

The disaggregation. The core transformation: instead of caching the merged $xW+xA_iB_i$ into one unified pool, ForkKV caches $xW$ (with RoPE) and $xA_i$ (without RoPE) separately.

Paper's Figure 8: "Unified v.s. Disaggregated KV Cache." (a) unified path merges base + LoRA and applies RoPE before caching; (b) disaggregated path stores bCache (RoPE-applied) and rCache (no RoPE, dimension mismatch defers it). The key structural point: RoPE lives on bCache but is deferred for rCache until reconstruction — this asymmetry propagates into the kernel design.

OS fork analogy. ForkKV frames cache creation as fork() + copy-on-write: the shared bCache = parent's read-only pages, the rCache = child's private CoW pages.

Paper's Figure 6: "ForkKV use OS-inspired fork semantics to create memory space for new agents with copy-on-write." (a) OS creating a child process vs (b) ForkKV creating a new agent. A new agent inherits the parent's bCache by mapping (Step 1) and only pays for its own rCache (Step 2), exactly like CoW page mapping.

End-to-end system. The serving pipeline threads a request from queue → scheduler → DualRadixTree prefix-match → agent runner (LoRA load + agent loop) → GPU executor + cache controller + ResidualAttention.

Paper's Figure 7: "Overview of ForkKV." Note the agent loop lives inside the agent runner — this is where model reasoning interleaves with external tool invocations, i.e. the actual agentic control loop that reuses the forked cache across turns.

The agent turn as a state machine. One agent turn cycles between reasoning (LLM generation over the forked cache) and acting (tool call), each act appending new tokens whose bCache stays shared and whose rCache is CoW-private:

Error / eviction recovery. Because base and residual pools have very different footprints and access frequencies, ForkKV uses a decoupled eviction policy: independent LRU per tree. If a large bCache node is evicted but its rCache survives, the request is a partial hit — recompute only the missing $xW$ base projection, reinsert into the base tree, and reuse the surviving $xA_i$. This is the fallback state when a cache slot is lost, avoiding a full recompute.

4. 作者证明 #

无形式化作者证明 — 仅实证. There is no convergence or accuracy guarantee. The authors explicitly concede that sharing bCache beyond the first layer is mathematically lossy (adapter activations make $x$ diverge across agents at each layer); the claim that this is bounded is empirical, backed by two mechanistic arguments and measured similarity, not a bound. What could have been bounded: a per-layer input-drift bound $\|x_l^{(i)}-x_l^{(j)}\|$ as a function of adapter magnitude and residual-connection contraction — the paper measures cosine similarity (>99.4%) but does not prove it.

Notation table (from the two load-bearing equations):

方程物理意义 (physical meaning of the equations):

• Reconstruction: $Y = bCache + rCache \times B_i$ — the full projection is the shared base plus the small residual up-projected once; this additivity is the algebraic license to physically split storage.
• Memory ratio: $M_R = \frac{sn + N\cdot sr}{N\cdot sn} = \frac{1}{N} + \frac{r}{n}$ — one shared base ($sn$) plus $N$ residuals ($sr$) versus $N$ full caches. As $N\to\infty$, $M_R\to r/n$; with $r=16,n=1024$ that is ~1.6% of unified memory.
• Attention fusion: $\sum sm(QK^{T})V = \sum sm(QK^{T})V_{base} + \big(\sum sm(QK^{T})V_{res}\big)B_v$ — associativity lets $B_v$ apply once to the accumulated residual scores instead of to every value block.

6 minimum reproduction checks:

1. Memory-ratio monotonicity. Verify $M_R=1/N+r/n$ decreases in $N$ and increases in $r$; matches the §3.2 example (16 agents, 32K, Llama3-8B: 64GB → ~5GB = 11.8×).
2. Size asymmetry. Confirm $n/r$ ratio: $n=1024,r=16 \Rightarrow$ bCache ~64× rCache; sanity-check against "64MB rCache vs 4GB bCache".
3. RoPE dimension check. rCache is $r$-dim, RoPE matrix $R_p$ is $n$-dim → RoPE cannot apply to rCache; it must be deferred to after $K_{res}B_k$. (Algorithm 1, line 8.)
4. Associativity fusion. Verify Eq.(4) numerically: eager $\sum sm(QK^T)(V_{base}+V_{res}B_v)$ equals fused $acc + acc_r\cdot B_v$ (Algorithm 1, lines 15–16, 20).
5. Quality-vs-similarity coupling. Reproduce the sweep: ForkKV keeps >99.4% input similarity → 1.60% max loss; full reuse drops to ~92.4% → 21.0% loss. Monotone: lower similarity ⇒ larger quality drop.
6. Online-softmax correctness. The kernel maintains running $m,l$ and rescales both $acc$ and $acc_r$ by $\exp(m-m_{new})$ (lines 15–16) — check that the residual accumulator is rescaled identically to the base accumulator, else the final $acc_r B_v$ term is misweighted.

Success-rate / sweep model (agent-specific). The paper's "success" axis is generation quality (F1), swept over (model × dataset × sharing policy) in Table 2 and (LoRA rank × output length) in Fig.15. Monotonicity: quality drop grows with task complexity (APIGen worst), throughput gain grows with memory contention (larger model, more workflows, higher arrival rate, longer outputs) and shrinks with LoRA rank (larger rank = larger rCache).

5. 实验与数据 #

Motivation — the bottleneck is real. Prefix caching cannot bridge distinct adapters, so throughput collapses as concurrent workflows grow.



Paper's Figure 3: "End-to-end throughput of prefix caching with different number of concurrent workflows." Throughput drops ~90% (1→8 workflows) because redundant per-agent caches exhaust GPU memory and starve batch parallelism — this is the collapse ForkKV targets.

The lossy-but-bounded claim. The central risk is that sharing bCache across layers degrades quality; the data says it barely does, while naive full reuse fails badly.



Paper's Figure 5: "(a) generation quality and (b) input x similarity compared to prefix caching" on APIGen. ForkKV holds >99.4% input similarity and only 1.60% quality loss; full reuse falls to ~92.4% similarity and 21.0% loss — the ~13× gap is entirely attributable to keeping the per-adapter rCache.

End-to-end throughput. The headline sweep across three models × three datasets × two workflow patterns.



Paper's Figure 11: "End-to-end throughput evaluation ... (tasks/s) of ForkKV against prefix caching baselines." Gains are largest where memory pressure is worst — 3.04× on Qwen2.5-14B (LooGLE, ReAct) but only 1.25× on the smaller Qwen2.5-7B, confirming ForkKV's value is proportional to contention.

Why it wins — the causal chain. The improvement decomposes into three measured levers.



Paper's Figure 14: "(a) average per-agent memory usage, (b) cache hit rate, (c) average decode batch size." Lower per-agent memory (12.7×) → higher hit rate (6.93×) → larger decode batch (12.0×). This is the load-bearing evidence that memory reduction, not kernel micro-optimization, drives throughput.

Accuracy verification (Table 2).

Model	Sharing Policy	HotpotQA	APIGen
Llama3-8B	Prefix Caching	57.63	39.77
Llama3-8B	ForkKV	57.17	38.17
Llama3-8B	Full Reuse	54.02	17.82
Qwen2.5-7B	Prefix Caching	57.14	92.28
Qwen2.5-7B	ForkKV	56.37	91.52
Qwen2.5-7B	Full Reuse	55.47	90.08
Qwen2.5-14B	Prefix Caching	70.91	94.56
Qwen2.5-14B	ForkKV	70.66	94.16
Qwen2.5-14B	Full Reuse	68.86	93.66

ForkKV sits just under the lossless Prefix-Caching upper bound everywhere (avg −0.71, max −1.60 on Llama3-8B/APIGen); Full Reuse craters on APIGen (39.77 → 17.82). Note the honest negative result in Fig.12: at light load (4 ReAct workflows) ForkKV is slower than baselines because the specialization overhead isn't amortized when memory is abundant — mitigated (proposed, not evaluated) by adaptive fallback scheduling.

6. 论证链 #

Step	Claim	Support (paper-internal)
1	Multi-LoRA agents share a massive static prefix but each adapter's activations make identical-text KV caches diverge, so prefix caching cannot share them.	§3.1; Fig.2 (ReAct/MapReduce reuse failure), Takeaway #1
2	Therefore per-agent redundant caches scale memory linearly with agent count, exhausting GPU memory and collapsing throughput (~90% drop, 1→8).	§3.1; Fig.1, Fig.3
3	LoRA's additive form $Y=xW+xA_iB_i$ permits splitting cache into shared bCache + tiny rCache; because $r\ll n$, bCache dominates size, so sharing it amortizes memory ($M_R=1/N+r/n$).	§2.2, §3.2, §5.1 Eq.(3); Fig.4
4	Cross-layer bCache sharing is mathematically lossy but empirically bounded by residual connections + preserved per-adapter QKV interactions (>99.4% similarity, 1.60% loss), unlike full reuse (92.4%, 21.0%).	§3.2, §7.3; Fig.5, Table 2
5	Realizing this needs (a) lifecycle management of 1-to-N base→residual mappings and (b) cheap reconstruction; solved by DualRadixTree + CoW fork and by ResidualAttention (SRAM reconstruction, deferred RoPE, associativity fusion).	§3.3, §4, §5.2, §5.3 Eq.(4), Algorithm 1; Fig.6, Fig.7, Fig.9, Fig.10
6	Result: 12.7× less per-agent memory → 6.93× hit rate → 12.0× decode batch → 1.25–3.04× throughput at ~0.71% quality cost.	§7.2; Fig.11, Fig.14, Table 2

7. 实现 cross-reference #

Built on SGLang v0.5.6, ~3K LoC Python + custom Triton kernels (§6). The public artifact is not linked in the source, so concrete file:line citations are unavailable — [实现未公开]. The design maps to these modules per §6:

• Disaggregation — a custom LoRA replacement module for the linear projection layer separates residual activations from base activations and stores rCache in a dedicated pool indexed by the residual RadixTree. (§6 "Disaggregated KV Cache")
• Control plane — SGLang's native RadixCache extended into the coordinated DualRadixTree; scheduler adapted to orchestrate the two-tiered pool across chunked prefill / non-chunked prefill / decode. (§6 "Control Plane and DualRadixTree Storage")
• Kernel — ResidualAttention in Triton, adapted from SGLang's RadixAttention, with separate prefill and decode versions. Algorithm 1 (L1 §5.3) is the reference pseudocode.

核心技术壁垒 (再述). The ResidualAttention kernel is the replication wall. Two things must be exactly right and are easy to get wrong: (1) deferred RoPE — because rCache is $r$-dimensional, RoPE is applied after the up-projection $K_{res}B_k$ inside the block loop (Algorithm 1 line 8: K_lora ← RoPE(K_res · B_k)), not at projection time; (2) matrix-associativity fusion — the up-projection $B_v$ must be lifted out of the inner loop and applied once at the end (acc_final ← acc + acc_r · B_v, line 20), which requires maintaining a separate residual accumulator acc_r that is rescaled by the online-softmax factor identically to acc (lines 15–16). Doing $V_{res}B_v$ eagerly per block instead negates the memory savings and destroys parallelism (the explicitly rejected naive path, §5.3).

关键实现细节 (easy-to-miss tricks).

1. Decoupled eviction with partial hit (§5.2): base and residual trees carry independent LRU state. A bCache miss while rCache survives triggers a partial recompute of only $xW$, reinserted into the base tree, reusing the surviving $xA_i$ — not a full cache miss. Coupling the two LRUs (cascading eviction) would force a low-contention pool to discard active cache and cause avoidable recomputation.
2. Residual tree key = token ids + agent id (§5.2): the base tree keys purely on token ids (enabling zero-copy sharing), but the residual tree extends the key with the agent id to isolate per-agent branches — miss this and agents would collide on each other's rCache.