ICaRus: Identical Cache Reuse across Models for Multi-Model LLM Serving
ICaRus: Identical Cache Reuse across Models for Multi-Model LLM Serving #
§1 TL;DR #
ICaRus decomposes decoder-only Transformers into a frozen logical encoder (KV generation) and task-specific logical decoders (next-token prediction), enabling identical KV cache sharing across multiple specialized models. With 8 agents it achieves 11.1× P95 latency reduction and 3.8× throughput gain while matching or exceeding fine-tuned accuracy.
§2 Q1 · Q2 · Q3 #
Q1 痛点 #
Multi-model inference — where task-specialized models (math, coding, reasoning) collaborate within agentic workflows — forces each model to maintain its own KV cache even when processing identical prompts. This causes three compounding problems:
- Memory explosion: KV cache memory grows as $\mathcal{O}(N)$ with model count $N$, saturating GPU memory far earlier than single-model serving.
- Eviction cascading: once GPU memory saturates, serving systems (vLLM, SGLang) evict cached prefixes → when the evicted model is invoked again, its entire prefix must be recomputed from scratch.
- No cross-model prefix caching: KV caches are model-specific, so even though multiple models see the identical prompt, each must independently run prefill. Prefix caching — the standard optimization for identical prefixes — is inherently per-model.
- Prefill: only the base model (logical encoder) runs; no adapter overhead. Generates KV cache and produces the first token.
- Decode: both encoder and decoder run in parallel. Encoder writes the new KV entry; decoder predicts the next token. Queries concatenated along head dimension for a single GQA call.
- Cross-model reuse: any subsequent request (from any model) hitting the same prefix reuses the existing KV cache. This is possible because all models share the identical encoder.
- Qwen3-8B shows consistent ICaRus advantage: +1.9 GSM8K, +1.4 GSM+, +4.9 HumanEval, +4.3 HumanEval+.
- LLaMA-3.1-8B math is the only clear loss case: 67.9 vs 69.7 on GSM8K.
- The "implicit regularization" effect appears model-dependent — stronger on Qwen3 than LLaMA.
- Query concatenation for parallel GQA: during decode, encoder and decoder queries are concatenated along the head dimension (shape $[2, 1, H, d_k] \to [1, 2H, d_k]$), fed through a single GQA attention call, then reshaped back. This avoids reading the KV cache twice and is the mechanism that keeps decode latency comparable to single-model despite 2× compute. The reshape-concat-reshape sequence must preserve head ordering to avoid mixing encoder and decoder attention outputs (Algorithm 3, Appendix B.2).
- Prefill = encoder only: during the prefill phase, no adapter weights are loaded or computed. The base model runs standard prefill and writes KV cache. This means prefill latency is identical to single-model serving — the 2× compute overhead only applies during decode. This asymmetry is easy to miss: ICaRus's prefill is strictly cheaper than baseline multi-model prefill (1× vs $N\times$), while its decode is at most 2× in compute but ~1× in memory access.
- Serving stage: both prefill and decode, with asymmetric behavior (prefill = encoder only, decode = encoder + decoder)
- Concurrency regime: evaluated at 2–8 concurrent agents; benefits grow with agent count
- Hardware affinity: evaluated on A100 80GB; memory-bound decode regime favors GPUs with high HBM bandwidth
- Ecosystem integration: implemented within vLLM; requires (a) model loading to separate encoder/decoder paths, (b) LoRA adapter hot-swapping for decoders, (c) modified attention kernel supporting concatenated queries. Migration cost: non-trivial vLLM modification (not a config flag)
- API: standard vLLM API; ICaRus is transparent to the client (same OpenAI-compatible endpoint)
- No DroidSpeak head-to-head comparison in experiments (only conceptual discussion)
- LoRA rank ablation missing — whether lower rank (e.g., 16 or 32) preserves accuracy is unknown
- Heterogeneous base model support (mixing LLaMA + Qwen) not evaluated
- Per-token decode latency overhead not empirically measured (only argued via memory-bound analysis)
Prior approaches attack only part of the problem: H2O / KVQuant / SwiftKV reduce single-model KV size; KVFlow schedules eviction/prefetching based on agent workflow but remains single-model; DroidSpeak shares non-sensitive layers between base and fine-tuned variants but must recompute sensitive layers.
Paper's Figure 1, verbatim (caption: "Comparison of KV cache management strategies and effectiveness in multi model scenarios between conventional approaches and ICaRus"). Sub-figure (a) shows how ICaRus eliminates per-model KV duplication by sharing a single cache; sub-figure (b) shows cross-model prefix caching enabling one prefill to serve all models.
Q2 方法 #
Core insight: a decoder-only Transformer can be decomposed into a logical encoder $E$ (generates KV caches) and a logical decoder $D$ (predicts the next token from the KV cache). In standard fine-tuning, both $E$ and $D$ are updated, making each model's KV cache unique. ICaRus freezes $E_{\text{base}}$ (the pretrained encoder) and fine-tunes only $D_{\text{task}}$ per task:
$$K_{1:i}, V_{1:i} = E_{\text{base}}(x_{1:i}) \quad\text{(frozen, shared across all tasks)}$$
$$x_{i+1} = D_{\text{task}}(x_i, K_{1:i}, V_{1:i}) \quad\text{(task-specific decoder)}$$
Since all task models share $E_{\text{base}}$, the KV cache for any given input is identical across models → direct sharing without approximation. Training explicitly accounts for the shared-KV setting: input is duplicated to both encoder and decoder, encoder generates KV, decoder attends to it and computes loss. This KV-sharing-aware training ensures robustness at inference time.
Inference optimization: during decoding, ICaRus concatenates query representations from the encoder and decoder along the head dimension and executes a single GQA attention call, reading the shared KV cache only once. Combined with shared base-model parameters, this keeps per-token latency comparable to a single model despite running both encoder and decoder.
核心技术壁垒: Freezing the entire logical encoder — half the model's parameters — does NOT degrade task accuracy. On Qwen3-8B/14B, ICaRus actually outperforms conventional fine-tuning (87.3 vs 85.4 on GSM8K for Qwen3-8B). The paper attributes this to "implicit regularization" from the frozen encoder: all task specialization must flow through the decoder, preventing overfitting. This counter-intuitive result is the core barrier to replication — one must believe (or verify) that encoder freezing is not merely lossless but beneficial.
Q3 结果 #
| Dimension | Result |
|---|---|
| Accuracy (Qwen3-8B, GSM8K) | ICaRus 87.3 vs baseline 85.4 (+1.9) |
| Accuracy (Qwen3-14B, GSM8K) | ICaRus 88.8 vs baseline 85.6 (+3.2) |
| Accuracy (LLaMA-3.1-8B, GSM8K) | ICaRus 67.9 vs baseline 69.7 (−1.8) |
| P95 latency (8 models, ReAct) | 11.1× reduction vs conventional |
| Throughput (8 models, ReAct) | 3.8× improvement vs conventional |
| P95 latency (swap-based, 8 models) | 12.1× reduction |
| Models validated | LLaMA-3.1-8B, Qwen3-1.7B/8B/14B/32B |
ICaRus wins on Qwen3 variants but loses slightly on LLaMA-3.1-8B math (67.9 vs 69.7). System gains grow super-linearly with agent count: throughput improvement from 1.4× (2 models) to 3.8× (8 models).
§3 架构 / 方法図 #
System scope #
| Dimension | ICaRus |
|---|---|
| Serving stage | Both prefill and decode |
| Serving mode | Continuous batching (vLLM integration) |
| Parallelism owned | None (orthogonal to TP/PP/EP) |
| Deployment | Single node (evaluated on 8×A100 80GB) |
| Adaptation method | LoRA (rank 128, α=256); agnostic to adapter type |
Architecture overview #
Paper's Figure 3, verbatim (caption: "Overview of the ICaRus architecture. The base model, a pretrained decoder-only Transformer, serves as the logical encoder, while the adapter-tuned model (consisting of the base model and a tunable adapter) serves as the logical decoder. The blue and orange lines indicate computations performed by the base model and the adapter-tuned model, respectively. The purple square denotes that the same base model generates the KV cache during both the prefill and decoding phases."). The figure shows how during decoding, the encoder and decoder share base-model parameters and the KV cache is written only by the encoder path. Queries from both paths are concatenated along the head dimension for a single GQA attention call.
Request lifecycle #
Key structural points:
§4 作者证明 #
Notation table #
| Symbol | Meaning |
|---|---|
| $F$ | Decoder-only Transformer (full model) |
| $E$, $D$ | Logical encoder, logical decoder |
| $E_{\text{base}}$ | Frozen pretrained logical encoder |
| $D_{\text{task}}$ | Task-specific fine-tuned logical decoder |
| $x_i$ | $i$-th token |
| $K_{1:i}$, $V_{1:i}$ | Accumulated key and value sets up to position $i$ |
| $N$ | Number of task-specific models (agents) |
| $M$ | Base model parameter count |
| $L_i$ | Input prompt length |
| $L_o$ | Output tokens per turn |
| $t$ | Interaction turns per adapter |
| $L_t = L_i + tL_o$ | Total sequence length |
Equations and physical meaning #
Eq 1 — Transformer as KV-conditioned predictor:
$$x_{i+1} = F(x_i, K_{1:i}, V_{1:i})$$
Next-token prediction depends only on the current token and accumulated KV cache, not on re-reading all previous tokens. This is the standard justification for KV caching.
Eq 2–3 — Encoder-decoder decomposition:
$$K_{1:i}, V_{1:i} = E(x_{1:i})$$
$$x_{i+1} = D(x_i, K_{1:i}, V_{1:i})$$
Any decoder-only Transformer $F$ can be split into an encoder $E$ (KV generation) and decoder $D$ (next-token prediction). The standard model is the special case where $E$ and $D$ share identical parameters.
Eq 4–5 — ICaRus sharing constraint:
$$K_{1:i}, V_{1:i} = E_{\text{base}}(x_{1:i})$$
$$x_{i+1} = D_{\text{task}}(x_i, K_{1:i}, V_{1:i})$$
Freezing $E_{\text{base}}$ guarantees that for a given input sequence, the KV cache is deterministic and model-independent. Multiple $D_{\text{task}}$ variants attend to the same cache.
Complexity model (Table 1) #
Paper's Table 1, verbatim (caption: complexity comparison between single model and multi model scenarios). ICaRus eliminates the $N$ factor from both memory and prefill, reducing multi-model overhead to single-model equivalence.
| Metric | Baseline (N models) | ICaRus | Reduction |
|---|---|---|---|
| Memory | $\mathcal{O}(M + NL_t)$ | $\mathcal{O}(M + L_t)$ | $N\times$ |
| Prefill latency | $\mathcal{O}(N(ML_t + L_t^2))$ | $\mathcal{O}(ML_t + L_t^2)$ | $N\times$ |
| Decode memory access | $\mathcal{O}(M + L_t)$ | $\mathcal{O}(M + L_t)$ | 1× |
| Decode compute | $\mathcal{O}(M + L_t)$ | $\mathcal{O}(2M + 2L_t)$ | 0.5× (2× cost) |
6 verification checks #
| # | Claim | Verification | Status |
|---|---|---|---|
| 1 | Eq 1–3 decomposition: $F = D \circ E$ | Standard KV cache semantics in decoder-only Transformers; $E$ extracts KV, $D$ uses it. The special case $E \equiv D$ recovers $F$. | ✓ valid |
| 2 | Eq 4: frozen $E_{\text{base}}$ ⇒ identical KV | Deterministic forward pass with fixed parameters and identical input yields identical output. Requires no stochastic layers (dropout=0 at inference). | ✓ valid |
| 3 | Memory $\mathcal{O}(M + L_t)$ eliminates $N$ | Single shared KV store serves all decoders; each decoder only adds lightweight adapter weights ($\ll M$). | ✓ valid |
| 4 | Prefill $\mathcal{O}(ML_t + L_t^2)$ eliminates $N$ | One prefill pass generates KV for all models; subsequent models skip prefill entirely via cache hit. | ✓ valid |
| 5 | Decode memory access $\mathcal{O}(M + L_t)$ despite 2× compute | Encoder and decoder share base parameters (loaded once) and read KV cache once via concatenated queries. Memory traffic ≈ single model. | ✓ valid in memory-bound regime; no per-token latency measurement provided |
| 6 | Training convergence: decoder-only fine-tuning matches full fine-tuning | Fig 2 shows loss curves overlap. "Implicit regularization" explanation is intuitive but lacks formal proof or ablation isolating the regularization effect. | ✓ empirically validated; theoretical gap |
§5 実験与数据 #
Accuracy across tasks and models #
Paper's Table 2, verbatim (caption: "Comparison of conventional methods and ICaRus on diverse datasets"). ICaRus achieves parity or better on Qwen3-8B across all tasks. On LLaMA-3.1-8B, ICaRus trails by 1.8 points on GSM8K but gains on GPQA (+1.5) and HumanEval+ (+2.4).
Key observations:
Scaling with model size #
Paper's Table 3, verbatim (caption: comparison across Qwen3-1.7B/8B/14B on MetaMathQA-40K). ICaRus advantage grows with model size: +0.8 at 1.7B, +1.9 at 8B, +3.2 at 14B on GSM8K.
P95 latency and throughput (main result) #
Paper's Figure 4, verbatim (caption: "P95 latency and throughput of ICaRus compared with multiple task-specific agents fine-tuned from the LLaMA-3.1-8B base model under the ReAct pattern"). Left: P95 latency across QPS for 2/4/8 agents. Right: throughput across QPS. ICaRus curves remain flat or improving while baseline degrades at moderate QPS due to KV eviction cascading.
Headline numbers at baseline's peak-throughput QPS:
| Agent count | P95 latency reduction | Max throughput gain |
|---|---|---|
| 2 | 3.8× | 1.4× |
| 4 | 5.1× | 2.3× |
| 8 | 11.1× | 3.8× |
The super-linear scaling with agent count is the key system result: each additional model in the baseline adds $\mathcal{O}(L_t)$ KV memory, accelerating memory saturation and eviction. ICaRus keeps KV at $\mathcal{O}(L_t)$ regardless of $N$.
Performance across diverse workflows and models #
Paper's Figure 5, verbatim (caption: "Comparison of P95 latency and maximum throughput across QPS for LLaMA3.1-8B and Qwen-3-14B Base under ReAct and Reflexion patterns"). ICaRus advantages generalize across model sizes (8B vs 14B) and agentic patterns (ReAct vs Reflexion). Qwen3-14B shows up to 7.4× latency reduction and 3.6× throughput gain.
Workload regime analysis #
| Workload regime | ICaRus | Baseline | Why |
|---|---|---|---|
| Many models ($N \geq 4$), moderate-high QPS | Large win (3.8–11.1× latency) | Memory saturates → eviction cascade | ICaRus eliminates $N\times$ KV growth |
| Few models ($N = 2$), low QPS | Moderate win (1.4× throughput) | KV fits in memory | Both fit; ICaRus saves prefix recompute |
| Skewed/random agent invocation | Large win (up to 15× P95 at $N = 2$) | Hot model evicts cold model's cache | ICaRus shares cache regardless of invocation order |
| Single model | No benefit | Standard serving | No cross-model sharing to exploit; decode overhead is pure cost |
| Short prompts, low concurrency | Minimal win | Low KV pressure | Less prefix to share |
§6 論証鏈 #
| Step | Claim | Evidence | Depends on |
|---|---|---|---|
| 1 | A decoder-only Transformer decomposes into logical encoder $E$ (KV generation) and logical decoder $D$ (next-token prediction). | Eqs 1–3: standard KV cache semantics. The special case $E \equiv D$ recovers $F$. | Transformer architecture definition |
| 2 | Freezing $E_{\text{base}}$ and fine-tuning only $D_{\text{task}}$ preserves task accuracy. | Table 2: ICaRus matches or exceeds conventional fine-tuning on 4/5 benchmarks (Qwen3-8B). Fig 2: training loss curves overlap. | Step 1 (decomposition must be valid) |
| 3 | Shared $E_{\text{base}}$ guarantees identical KV caches across models for identical inputs. | Eq 4: deterministic forward pass with frozen parameters. No approximation or layer selection needed (unlike DroidSpeak). | Step 2 (encoder must be frozen) |
| 4 | Identical KV caches enable cross-model prefix caching and eliminate $N\times$ memory/prefill overhead. | Table 1: memory $\mathcal{O}(M + L_t)$ vs $\mathcal{O}(M + NL_t)$; prefill latency eliminates factor $N$. | Step 3 (KV identity guarantee) |
| 5 | Query concatenation along head dimension enables parallel encoder–decoder execution with single KV read. | Fig 3, Algorithm 3 (Appendix B.2): concatenated queries → single GQA call. Memory access $\mathcal{O}(M + L_t)$ matches single-model. | Step 3 (shared KV) + GQA mechanics |
| 6 | End-to-end system gains grow super-linearly with agent count. | Fig 4: 1.4×/2.3×/3.8× throughput at $N$ = 2/4/8. Baseline throughput degrades at lower QPS as $N$ increases (eviction onset earlier). | Steps 4 + 5 (memory + latency benefits compound) |
§7 実現 cross-reference #
[実現未公開] — the paper does not release source code. Implementation details are described at pseudocode level in Appendix B (Algorithms 1–3) and the system is evaluated within vLLM, but no repository URL or patch is provided.