NetKV: Network-Aware Decode Instance Selection for Disaggregated LLM Inference #

§1 TL;DR #

NetKV adds datacenter network topology and congestion awareness to decode-instance selection in disaggregated LLM inference via a lightweight operator-to-scheduler oracle; on a 64-GPU fat-tree, it cuts mean TTFT by up to 21% over round-robin and 18% over cache+load-aware baselines, with the static tier map alone capturing >90% of the gain.

§2 Q1 / Q2 / Q3 #

Q1 痛点 #

Disaggregated LLM inference separates prefill and decode onto distinct GPU pools for efficiency, but this forces the KV cache (up to 40 GB for 128K-context Llama-3-70B) to traverse the datacenter network before decoding begins. Transfer time directly enters the TTFT budget. Existing schedulers (Mooncake Conductor, llm-d, Dynamo) route based only on compute load and prefix-cache locality — they are blind to network topology and dynamic congestion. A decode instance with 90% cache hit on a congested cross-pod link can yield worse TTFT than a cold-cache same-rack instance.

The core information asymmetry: the inference scheduler has no visibility into physical topology or link utilisation; the network operator has no knowledge of upcoming KV transfers. Neither side alone can make optimal placement decisions.

Q2 方法 #

Network Cost Oracle — a thin interface between operator and scheduler exposing four maps refreshed every $\Delta_{\text{oracle}}$ seconds:

1. tier_map: instance pairs → {0,1,2,3} (static, from K8s topology labels)
2. tier_bandwidth: tier → Gbps (static, hardware specs)
3. tier_latency: tier → μs (static)
4. congestion: tier → [0,1) (dynamic, from switch telemetry)

Algorithm: $O(|\mathcal{D}|)$ per-request greedy scoring each candidate decode instance by the sum of three costs:

$$C(d) = T_{\text{transfer}}(p, d, s_r^{\text{eff}}(d)) + T_{\text{queue}}(d) + T_{\text{decode}}(d)$$

where effective bandwidth accounts for static tier bandwidth, external congestion, and self-contention:

$$B_{\text{eff}}(p,d) = \frac{B_{\tau(p,d)} \cdot (1 - c_{\tau(p,d)})}{1 + n_{\text{inflight}}^{\tau}(p)}$$

核心技术壁垒: The dominant insight is that static topology tier information alone (requiring only existing Kubernetes labels, zero dynamic telemetry) captures >90% of the scheduling benefit. This renders deployment trivially simple — the oracle's "dynamic" component is nearly unnecessary, yet the paper's formal framework proves the tier-ranking remains robust to 42 percentage points of staleness error (Proposition 2). The combination of theoretical elegance and deployment minimalism is the hardest insight to replicate without the paper's analysis.

Q3 结果 #

On a 64-GPU four-tier fat-tree simulator with Mooncake production traces:

• Mean TTFT: −21.2% over round-robin, −17.6% over CLA* (tuned cache+load-aware)
• SLO attainment: +20.1 pp (from 79.1% to 99.2% at 16K tokens)
• TBT overhead: <0.5 ms in all conditions (order of magnitude below interactive SLOs)
• Transfer time: −25.7% (835→620 ms) via tier-shifting from cross-pod to intra-pod
• Scalability: advantage persists to 1024 GPUs; scheduler decision latency <1.5 ms

---

§3 架构 / 方法図 #



Paper's Figure 1, verbatim (caption: "NetKV-Full mean-TTFT reduction over CLA (%) across the topology sweep for each workload profile").*

This heatmap demonstrates that NetKV-Full wins in all 60 topology × workload cells. The reduction grows monotonically along both axes — oversubscription ratio (rows) and background traffic intensity (columns) — confirming the algorithm's network sensitivity is well-calibrated: it activates precisely when the network bottleneck is acute.

System Architecture (Mermaid) #

flowchart TB subgraph Operator["Network Operator"] SW["Switch Telemetry
(INT/sFlow/SNMP)"] TOPO["Topology DB
(K8s labels)"] end subgraph Oracle["Network Cost Oracle (refreshed every Δ_oracle)"] TM["tier_map: (p,d) → {0,1,2,3}"] TB["tier_bandwidth: tier → Gbps"] TL["tier_latency: tier → μs"] CG["congestion: tier → [0,1)"] end subgraph Scheduler["NetKV Scheduler (per-request, O(|D|))"] FE["Feasibility Filter
(memory check)"] SC["Score each d ∈ D_r:
C(d) = T_xfer + T_queue + T_decode"] SEL["argmin C(d) → d*"] NI["n_inflight counter
(self-contention)"] end subgraph Cluster["64-GPU Fat-Tree Cluster"] subgraph Pod1["Pod 1"] subgraph Rack1["Rack 1"] P1["Prefill Instance p₁
(TP=4, 4 GPUs)"] D1["Decode Instance d₁"] end subgraph Rack2["Rack 2"] D2["Decode Instance d₂"] end end subgraph Pod2["Pod 2"] subgraph Rack3["Rack 3"] D3["Decode Instance d₃"] end end end SW --> CG TOPO --> TM TOPO --> TB TOPO --> TL Oracle --> Scheduler P1 -->|"KV cache transfer
(tier τ)"| D1 P1 -.->|"cross-pod path
(avoided by NetKV)"| D3 FE --> SC --> SEL --> NI SEL -->|"dispatch to d*"| D1

Four-Tier Fat-Tree Model #

Tier	Locality	Bandwidth	Latency	Technology
0	Same node	450 GB/s	1 μs	NVLink
1	Same rack	100 Gbps	3 μs	RoCE via ToR
2	Same pod	50 Gbps (2:1 ovsub)	8 μs	Spine hop
3	Cross pod	25 Gbps (4:1 ovsub)	15 μs	Core layer

Algorithm 1 Pseudocode (NetKV Decode Instance Selection) #

Input: request r, prefill instance p, decode pool D, oracle O
Output: selected decode instance d*

1. D_r ← {d ∈ D | m_d ≥ s_r^eff(d) + m_min}     # memory feasibility
2. for each d ∈ D_r:
3.   τ ← O.tier_map(p, d)                          # static lookup
4.   B_eff ← O.B[τ] · (1 - O.c[τ]) / (1 + n_inflight[τ][p])
5.   λ ← block_prefix_match(h_r, K_d)             # cache hit
6.   s_eff ← s_r · (1 - λ/ℓ_r)                    # effective payload
7.   T_xfer ← s_eff / B_eff + O.L[τ]
8.   T_queue ← max(0, q_d - (β_max - β_d)) · t_iter(β_d)
9.   T_decode ← t_iter(β_d + 1)
10.  C[d] ← T_xfer + T_queue + T_decode
11. d* ← argmin C[d]
12. n_inflight[τ(p,d*)][p] += 1
13. return d*

---

§4 作者证明 #

符号表 #

Symbol	Meaning	Typical value
$\tau(p,d)$	Locality tier between instances $p$ and $d$	{0,1,2,3}
$B_\tau$	Static tier bandwidth	450 GB/s / 100 Gbps / 50 Gbps / 25 Gbps
$c_\tau$	Per-tier congestion factor	[0, 1)
$n_{\text{inflight}}^\tau(p)$	Concurrent KV transfers from $p$ on tier $\tau$	0–16
$s_r$	Total KV cache size	$2 \cdot n_L \cdot n_{kv} \cdot d_h \cdot \ell_r \cdot b$
$s_r^{\text{eff}}(d)$	Effective transfer after cache hit	$s_r \cdot (1 - \lambda_r(d)/\ell_r)$
$\lambda_r(d)$	Block-aligned prefix hit length	tokens
$\epsilon$	Max oracle staleness error	$< 0.42$ for 4:1 topology

方程物理意义 #

Eq. (1) — KV cache size: $s_r = 2 \cdot n_L \cdot n_{kv} \cdot d_h \cdot \ell_r \cdot b$ directly computes the transfer payload from model architecture parameters. Factor 2 for K and V tensors. Linear in sequence length — this is why longer contexts make the network bottleneck acute.

Eq. (4) — Effective bandwidth: $B_{\text{eff}} = B_\tau(1-c_\tau)/(1+n_{\text{inflight}}^\tau)$ composes residual-bandwidth approximation (TCP/RDMA fluid model) with max-min fair sharing (DCQCN steady state). The two reduction factors are multiplicative and independently meaningful.

Eq. (5) — Objective: $C(d) = T_{\text{transfer}} + T_{\text{queue}} + T_{\text{decode}}$ is a serial model capturing all three latency components from prefill completion to first token emission.

6 Minimum Checks #

1. Dimensional consistency (Eq. 1): $[2 \cdot \text{layers} \cdot \text{heads} \cdot \text{dim} \cdot \text{tokens} \cdot \text{bytes}] = \text{bytes}$ ✓. For Llama-3-70B: $2 \times 80 \times 8 \times 128 \times 1 \times 2 = 327{,}680$ bytes/token = 320 KB/token ✓ (matches paper).
1. Worked example verification (§III-D): $s_r^{\text{eff}}(d_1) = 10\text{ GB} \times (1 - 0.5) = 5$ GB; $B_{\text{eff}}(d_1) = 6.25 \times 0.8 / 2 = 2.5$ GB/s; $T(d_1) = 5/2.5 = 2.0$ s ✓. For $d_2$: $s_r^{\text{eff}} = 10 \times 0.1 = 1$ GB; $B_{\text{eff}} = 3.125 \times 0.8/1 = 2.5$ GB/s; $T(d_2) = 0.4$ s ✓.
1. Proposition 1 bound check: With $\rho_1=0$, $\rho_2=0.5$, $k=4$, equal congestion, equal queues: LHS $= 1-0 = 1$; RHS $= 4 \times (1-0.5) = 2$. Since $1 < 2$, $d_1$ wins ✓.
1. Proposition 2 tolerance: $\epsilon < (B_\tau(1-c_\tau^) - B_{\tau'}(1-c_{\tau'}^))/(B_\tau + B_{\tau'})$. At $B_1/B_3 = 4$, $c = 0.3$: $\epsilon < (4 \times 0.7 - 1 \times 0.7)/(4+1) = 2.1/5 = 0.42$ ✓.
1. Complexity claim: Algorithm iterates once over $|\mathcal{D}_r|$ candidates with constant-time tier lookup + arithmetic per candidate → $O(|\mathcal{D}_r|)$ ✓.
1. Bandwidth budget (cluster-specific): At Tier 2 (50 Gbps = 6.25 GB/s), transferring 5 GB effective payload with no congestion and one inflight flow: $T = 5/3.125 = 1.6$ s. Paper reports 620 ms mean transfer (mix of partially-cached transfers) — consistent with average effective payload $\approx 1.5$ GB at mean $B_{\text{eff}} \approx 2.5$ GB/s → $0.6$ s ✓.

---

§5 实验与数据 #

Experiment 1: Load Sweep (Table II — RAG profile) #

Rate	Scheduler	TTFT (ms)	TBT (ms)	SLO	Transfer (ms)
100%	RR	1969±9	12.74	0.907	993
100%	CLA*	1812±15	12.71	0.923	835
100%	NetKV-Full	1598±10	12.94	0.944	620
200%	RR	2171±4	13.01	0.887	1194
200%	CLA*	1995±25	12.88	0.906	1018
200%	NetKV-Full	1710±9	13.29	0.933	733

Peak improvement: −21.2% TTFT over RR at 200% load. TBT overhead stays <0.5 ms — an order of magnitude below practical SLO thresholds.

Experiment 2: Context Length Sweep (Table III) #

Length	Δ TTFT vs RR	Δ TTFT vs CLA*	Δ SLO vs RR
1024	−9.1%	−2.2%	0.000
4096	−13.3%	−3.2%	0.000
8192	−15.2%	−10.2%	0.000
16384	−20.2%	−17.6%	+0.201
32768	−6.7%	−4.3%	−0.006
65536	+0.0%	+0.2%	0.000

Peak at 16K tokens — the "sweet spot" where KV cache is large enough for inter-tier bandwidth gap to matter (5 GB aggregate) yet small enough for requests to remain within SLO. Beyond 32K, all schedulers fail uniformly.



Paper's Figure 2, verbatim (caption: "Oracle staleness sweep: TTFT, TBT, and SLO are invariant from 100 ms to 60 s refresh intervals").

This result validates Proposition 2 empirically: all three metrics remain flat across three orders of magnitude of refresh interval. The practical implication is that a standard SNMP poll at once-per-minute cadence is sufficient — the telemetry burden on operators is near zero.



Paper's Figure 4, verbatim (caption: "Ablation ladder: mean TTFT for CLA, NetKV-Topo-Only, NetKV-Static, and NetKV-Full across the chatbot, RAG, and long-context workloads").*

The ablation clearly shows the static tier map delivers the lion's share of improvement (10.2% on RAG), self-contention adds 1.9%, and dynamic congestion contributes only 0.3%. This decomposition is the paper's most actionable finding for practitioners: deploy the minimal version first.

Experiment 6: Ablation (Table IV — incremental contribution) #

Component	RAG TTFT	Δ	Long-ctx TTFT	Δ
CLA*	1812 ms	—	7121 ms	—
+ Static tier map	1627 ms	−10.2%	6326 ms	−11.2%
+ Self-contention	1596 ms	−1.9%	6150 ms	−2.8%
+ Dynamic congestion	1592 ms	−0.3%	6160 ms	+0.2%

Dynamic congestion on long-context actually regresses by +0.2%, suggesting the signal can hurt in some regimes by causing over-reaction to transient congestion when transfers are already very long.

Experiment 5: Prefix Sharing Interaction (Figure 3) #



Paper's Figure 3, verbatim (caption: "Prefix-sharing sweep on the RAG workload: NetKV-Full preserves a roughly constant TTFT advantage over CA and CLA across the full range, indicating that the network-aware contribution is orthogonal to the cache-aware contribution").*

Across $p_{\text{share}} \in [0.0, 0.9]$, NetKV-Full maintains a 15.3–15.6% TTFT reduction over both CA and CLA*. The flat advantage line demonstrates that network awareness is orthogonal to cache awareness — cache hits reduce the payload equally on any tier, so the relative bandwidth advantage of intra-pod routing persists regardless of hit ratio.

Tier-Shifting Mechanism (Table VI) #

Tier	CLA*	NetKV-Full
Tier 2 (same-pod)	32.0%	68.9%
Tier 3 (cross-pod)	68.0%	31.1%
Mean transfer time	835 ms	620 ms

CLA* sends 68% of traffic cross-pod (the slowest path) because it is topology-blind. NetKV reverses this ratio to 69:31 intra-pod, directly causing the 25.7% transfer time reduction.



Paper's Figure 5, verbatim (caption: "Scalability: mean TTFT, mean TBT, SLO attainment, and scheduler decision latency from 64 to 1024 GPUs").

CLA*'s transfer time rises with scale (more cross-pod routing at larger clusters), while NetKV-Full keeps transfer time flat by maintaining topology locality. Scheduler decision latency stays sub-linear, remaining below 1.5 ms even at 1024 GPUs — confirming the $O(|\mathcal{D}|)$ algorithm is practical at production scale.

---

§6 论证链 #

Step	Claim	Evidence	Depends on
1	KV cache transfer is the dominant TTFT component in disaggregated inference at long context	Llama-3-70B at 128K: 40 GB KV, 3.2 s transfer on 25 Gbps link dominates any reasonable TTFT budget (§I)	§III-B Eq. 1
2	Current schedulers are provably suboptimal because they ignore network topology	Proposition 1 (§IV-C): network-oblivious scheduling is arbitrarily worse as $\ell_r$ grows; numerical example shows 5× transfer time penalty	Step 1, §III-D Eq. 3-4
3	A thin oracle interface suffices to expose the missing signal	Oracle needs only 4 maps; static components from existing K8s labels, dynamic component from standard switch telemetry (§III-E)	§III-A tier model
4	NetKV's $O(	\mathcal{D}	)$ algorithm is robust to oracle staleness	Proposition 2 (§V-D): tier ranking tolerates 42 pp error before inversion; empirically flat from 100 ms to 60 s refresh (§VI-F, Fig 2)	Steps 2-3, §V-D
5	Static topology signal is the dominant component	Ablation (§VI-H, Table IV): static tier map alone captures 10.2% of 11.9% total RAG improvement (>85%). Dynamic congestion adds ≤0.3%	Step 4, §VI-H
6	Mechanism is tier-shifting: redirecting transfers from cross-pod to intra-pod	Table VI: CLA* routes 68% cross-pod → NetKV routes 69% intra-pod; mean transfer drops 835→620 ms (§VII-B)	Steps 3-5
7	Benefits scale with network stress and cluster size	Topology sweep (Fig 1): gains grow with oversubscription and background traffic; scalability (Fig 5): advantage persists to 1024 GPUs with flat transfer time	Steps 5-6, §VI-E/I

---

§7 实现 cross-reference #

[实现未公开] — the paper is simulation-only with no released implementation.

However, the paper provides concrete integration targets:

• llm-d integration point: pluggable scorer chain in the Endpoint Picker. Interface: Score(ctx, pods) → map[Pod]float64. NetKV would be one scorer alongside existing prefix-cache, load, and session-affinity scorers.
• Dynamo integration point: KV-aware router's scoring function.
• Self-contention counter: decremented via vLLM's KVConnectorBase_V1.get_finished(finished_req_ids) — the existing transfer-complete API that engines already use to release prefill-side buffers. No new transport notification required.
• Static oracle sources: topology.kubernetes.io/zone labels + rack labels (already standard in K8s clusters).
• Dynamic congestion: per-tier aggregation of In-band Network Telemetry / sFlow / SNMP counters.

关键实现细節 #

1. $n_{\text{inflight}}$ counter cap: default 16 (roughly the NIC's saturated flow count). Without this cap, the self-contention denominator grows unboundedly and collapses effective bandwidth estimates to near-zero for bursty prefill instances.
1. DSCP separation for telemetry: the operator's congestion signal must exclude the scheduler's own KV flows (via marked DSCP class or dedicated QoS queue) to avoid double-counting with $n_{\text{inflight}}$. When this separation is unavailable, the fallback is $n_{\text{inflight}} \equiv 0$ with $c_\tau$ absorbing everything — graceful degradation.

---

§8 Cluster-Specific: Topology & Congestion Analysis #

System Scope #

• Scale: Single datacenter, rack/pod/cross-pod granularity (4-tier fat-tree)
• Workload: Inference (disaggregated P2P KV transfers, not collective communication)
• HW class: 100 Gbps RoCE (Tier 1), NVLink intra-node (Tier 0), oversubscribed Ethernet fabric (Tier 2/3)

Congestion Control Context #

• Model: Residual-bandwidth approximation from DCQCN convergence (max-min fair sharing)
• Load balancing: ECMP (uniform random uplink assignment at flow start); per-tier aggregation is exact for Tier 0/1 but approximate for Tier 2/3 where ECMP spreads flows across multiple spine links
• Refresh: Oracle polls at $\Delta_{\text{oracle}} \geq 1$ s — far longer than DCQCN convergence horizon, so steady-state approximation is appropriate

Software → Hardware Reverse Implication #

The paper implicitly argues for:

• Per-flow QoS marking (DSCP classes) to separate KV transfer flows from background traffic in telemetry
• Topology-aware pod labels as first-class scheduling signals (already available in K8s but underutilised by inference schedulers)
• Standard switch telemetry exposure (sFlow/INT) at the per-tier aggregate level — no per-flow visibility needed

---

§9 Open Questions #

1. Real-cluster validation gap: All results are simulation-only. The flow-level simulator overestimates gains by 4–6 pp versus packet-level (Table V). Real NIC-level effects (driver queueing, completion-queue contention, ECMP hash collisions) may further erode the advantage.
1. Tiers 0/1 utilisation is 0% in all experiments (Table VI) — because placement never co-locates prefill and decode at server/rack granularity. Would a mixed-placement strategy (some decode on same rack as prefill) capture the 450 GB/s NVLink tier?
1. KV compression impact: FP8/INT4 KV cache (emerging in production) halves or quarters $s_r$, proportionally reducing the network term's weight. At what compression ratio does NetKV's advantage become negligible?
1. MoE extension: For expert-parallel models, TBT itself becomes network-sensitive (All-to-All every step). Joint TTFT/TBT optimisation is a qualitatively different problem.
1. Multi-tenant price-of-anarchy: When multiple tenants independently make locality-biased decisions, does the system converge to a Nash equilibrium worse than the cooperative optimum?
1. Pipelining interaction: The serial transfer model is conservative. Under Splitwise/Dynamo-style pipelining that overlaps transfer with decode, the visible $T_{\text{transfer}}$ shrinks — potentially narrowing NetKV's advantage in practice.

---

§10 Deployment Context #

• Deployer profile: Any cloud provider or enterprise running disaggregated LLM inference at ≥32 GPUs with multi-tier network fabric
• Greenfield vs retrofit: Pure overlay — requires no hardware, transport, or engine changes. Deploys as a scoring plugin in existing scheduler (llm-d / Dynamo)
• Minimal viable deployment: Static tier map only (from K8s labels) captures >90% of benefit; dynamic telemetry is optional enhancement
• Vendor independence: No specific NIC/switch vendor dependency; works with any RDMA-capable fabric exposing standard topology labels and switch counters