ELDR: Expert-Locality-Aware Decode Routing for PD-Disaggregated MoE Serving — L2 #

1. TL;DR #

In xPyD MoE serving, decode latency is set by the union of distinct experts a batch loads from HBM, not by request count. ELDR reads each request's prefill expert activations into a compact signature, clusters them offline with balanced K-means (one centroid per decoder), and routes online within a similarity band to the least-loaded matching decoder — cutting median TPOT 5.9–13.9% losslessly.

2. Q1/Q2/Q3 #

System scope (framework-specific pin-down). ELDR is a serving framework, decode-side only, operating in a PD-disaggregated (xPyD) deployment. It owns the decode-worker routing decision at the prefill→decode handoff; it delegates prefill routing (PrefixHash affinity), batching (continuous batching, unchanged), and kernels to the underlying vLLM stack. Parallelism axes: it is agnostic to TP=1 single-GPU decoders in the main study, but composes with EP (expert parallelism) for the 235B case via per-decoder expert placement. Deployment mode is strictly multi-node disaggregated (up to 40 GPUs / 5 nodes).

Q1 — 痛点 (pain point). Existing decode routers balance load only (Random, RR, JSQ, P2C) and treat decode workers as interchangeable. That is correct for dense models — equal-load workers do equal FFN work — but wrong for MoE. Decode is memory-bandwidth bound, and each step must fetch weights for every distinct expert any token in the batch selects. Sparsity inverts: routing each token to few experts fragments the batch and destroys the weight reuse a dense batch enjoys. So the per-step union of experts, invisible to load-only routers, is a first-order latency knob. On Qwen3-30B-A3B growing active experts 16→128 raises MoE-layer latency $4.7\times$ at fixed batch size, while batch size at fixed expert count barely moves it.

Q2 — 方法 (method). ELDR adds a second routing axis: expert locality, exploiting three empirical facts: (1) experts specialize by domain, so same-domain batches activate 17–21% (task) / 3–10% (language) fewer distinct experts; (2) prefill and decode per-expert activation correlate at 0.70–0.92, so the signal is visible at handoff, before any decode token; (3) it's cheap to capture as a by-product of the gate. The mechanism has three parts: an expert signature (IDF-reweighted, layer-masked, L2-normalized discrete top-$k$ count) whose cosine distance predicts decode-time overlap; balanced K-means + locality-band routing that splits the locality (offline, aggregate) and load (online, instantaneous) objectives along their information boundary; and a block-granular signature cache co-indexed with the KV cache that keeps signatures exact under prefix caching.

核心技术壁垒 (the single hardest-to-replicate insight): the decoupling of signature quality from downstream clustering via a single scalar objective $\rho$ = Spearman rank correlation between signature-pair-distance and decode-pattern-pair-distance (Eq. 1). This lets ELDR pick a representation (discrete count vs. gate-prob vs. logit vs. binary), an IDF reweighting, and a layer mask independently of and before committing to any router — and crucially it validates the model first-order (each choice must move $\rho$, then $\rho$ must predict serving-TPOT $\Delta$, Fig. 14) rather than sweeping the full serving system per candidate. Reproducing ELDR without this offline predictor forces an expensive end-to-end sweep over the joint (representation × clustering × $\tau$) space.

Q3 — 结果 (results). Implemented in vLLM (~2,000 LoC Python) on AMD MI300X, up to 40 GPUs. Over the best of four load balancers: median TPOT −7.0–13.9% (task), −5.9–10.0% (language); tail TPOT −3.4–6.0% (task). Serving overhead 0.86 ms/req (1.2% of 69 ms TTFT), signature cache <1% of KV cache. Outputs are bit-identical to standard top-$k$ gating. Generalizes to Qwen3-235B-A22B at TP=4/EP=4 (−2.7–4.3% median TPOT) and improves monotonically with decoder-pool size (8.0%→9.8%→10.2% from 8P8D→8P24D).

3. 架构 / 方法图 #

Paper's Figure 6, verbatim (caption: "ELDR architecture: offline fitting of one centroid per decode worker over expert signatures, then online routing at the prefill→decode handoff by signature similarity, subject to load.").

This is the load-bearing architecture figure. It shows ELDR sitting as a thin router in front of the prefill/decode worker pools, with an offline stage (calibration signatures → balanced K-means → one centroid per decoder) feeding a centroid table, and an online stage that, at each handoff, compares a request's signature to those centroids and selects a decoder under a load constraint. Note that the model, kernels, batching, and the prefill/decode engines are untouched — the entire mechanism lives in the router plus a prefill-time capture hook.

Request/task lifecycle (framework §3 requirement — scheduler, KV manager, cross-node path):

• Scheduler (first-class box): the router is the scheduler; queueing discipline is locality-band JSQ — a τ-bounded shortest-queue over the workers matching the signature. Batching decisions inside a decoder are the stock continuous batching, unmodified.
• KV / memory manager: paged KV cache at block granularity; ELDR's signature cache is co-indexed at the same block id — allocation unit = one KV block, one signature row per block (int8 per (block, layer, expert)).
• Cross-node communication: KV transfer over vLLM's NIXL connector, 400 Gbps NDR InfiniBand (8 ConnectX-7 HCAs/node); the signature (12 KiB) piggybacks on the existing handoff metadata — no new RPC channel.

4. 作者证明 #

The paper has a light formalization — two numbered equations defining the signature and its quality metric, plus a memory-cost bound. There is no closed-form throughput/latency model (no λ/μ/B queueing model). Below is the notation table, physical meaning, and the six minimum checks.

Notation table:

Equation 1 — signature quality (the objective everything is fit against):

$$\rho = \mathrm{Spearman}\big(\mathrm{cos\text{-}dist}(s_i, s_j),\ \mathrm{cos\text{-}dist}(p_i, p_j)\big)$$

Physical meaning: how faithfully the ordering of signature pair-distances matches the ordering of true decode-pattern pair-distances. Why Spearman not Pearson: signature distance lives in $\mathbb{R}^d$, decode-pattern distance in $\mathbb{R}^{LE}$ — numerically incommensurable, only their ordering is comparable, and ordering is exactly what the clustering layer consumes.

Equation 2 — signature normalization:

$$s_r = x_r / \lVert x_r \rVert_2, \quad x_r = \big[\tilde{c}_r(\ell)\big]_{\ell \in \mathcal{S}}$$

Physical meaning: dividing by the vector length makes similarity reflect the shape of expert usage, not the prompt's token count, so long and short prompts of the same domain share a signature direction.

IDF weight (inline load-bearing): $w(\ell,e) = \log\big((|\mathcal{C}|+1)/(\mathrm{df}(\ell,e)+1)\big)$, reweighted count $\tilde{c}_r(\ell,e) = c_r(\ell,e)\cdot w(\ell,e)$. Physical meaning: generalist experts fire on nearly every request, inflate the norm, and mask rare specialists; IDF shrinks common, amplifies rare.

Block-sum (prefix-cache coherence): $s_r = \sum_{b\in\mathcal{B}(r)} \mathrm{sig}[b]$ — exact regardless of which request populated each block.

Six minimum checks:

1. Units / dimensionality. $s_r \in \mathbb{R}^{N^\cdot E}$ (masked layers × experts); $p_i \in \mathbb{R}^{LE}$ (all layers × experts). They differ deliberately — the signature is compressed; Eq. 1 only compares distances*, never the vectors directly, so the dimension mismatch is legitimate.
2. Boundary — $\tau$. $\tau=0$ ⇒ band = single nearest centroid ⇒ pure top-1 (max locality, no load slack); $\tau=1$ ⇒ band = all workers ⇒ pure shortest-queue. The optimum is interior ($\tau=0.1$): Fig. 16 shows $\tau=0$ regresses tail on 4/6 workloads, and beyond 0.1 median erodes as requests spill their band.
3. Monotonicity — decoder pool. Improvement is monotone increasing in $K$ (8.0→9.8→10.2%, Table 2): more decoders ⇒ finer clusters ⇒ narrower per-decoder union. What breaks it: at very large $K$ each cluster's calibration volume thins, but the study stops at $K=24$ before that.
4. Why balance constraint. Vanilla K-means has no size penalty ⇒ dense-region centroids absorb all points ⇒ uneven decoder load ⇒ tail spikes (17.4% tail regression, Fig. 15). The $\lceil N/K\rceil$ cap via Hungarian assignment forces centroids to cover sparse regions too.
5. Memory bound. Signature cache $\leq \text{num\_gpu\_blocks}\cdot L\cdot E$ bytes (int8/cell). For $L\leq48, E=128$: ≤6 KiB/block ⇒ <1% of KV cache. First-order check: 0.24% HBM reported (§6.2) is consistent with this bound.
6. Losslessness. ELDR changes only which decoder serves a request; the gate, top-$k$ selection, kernels, and batching are untouched ⇒ expert selections and outputs are identical to standard gating. No accuracy check needed — it is exact by construction.

5. 实验与数据 #



Paper's Figure 2 (caption: "MoE layer latency scales with active experts, not batch size (single MoE layer, one MI300X).").

The motivating microbenchmark: sweeping active-expert count vs. batch size on a single MI300X. Latency tracks active experts far more strongly — the 16→128 sweep is a $4.7\times$ swing at batch 64 while batch size at fixed expert count is nearly flat. This is the empirical foundation for treating the expert union as the latency knob.



Paper's Figure 4 (caption: "Same-domain batches (blue) activate fewer experts per decode step than mixed-domain (orange), for task (top) and language (bottom) across three MoE models.").

Quantifies the exploitable structure: same-domain batches activate 17–21% (task) / 3–10% (language) fewer experts. Note the language gap is much smaller — foreshadowing why the Domain baseline collapses on language while ELDR's finer clusters still win.



Paper's Figure 11 (caption: "TPOT (median, p99) and median TTFT vs request rate on the task workload at 8P16D.").

The main task result. ELDR sits below every baseline at every rate for median TPOT (−7.0–13.9%). Domain is a strong baseline here because task labels align with expert clusters, yet ELDR still beats it (finer $K=16$ clusters + $\tau$-band spill across cluster boundaries Domain's hard partition forbids). TTFT tracks baselines, dropping near saturation as faster decoders relieve prefill back-pressure.



Paper's Figure 15 (caption: "Mean % Δ vs. RR over five request rates (20–100 qps) at 8P16D with τ=0.1.").

The key ablation justifying the balance constraint. Vanilla K-means gives up to −9.8% median but +17.4% tail — the locality win is real but load imbalance blows the tail past round-robin. Hungarian-balanced K-means recovers both (−12.6% P50, −6.8% P99). This is not a minor refinement; it is the difference between a usable and unusable policy.



Paper's Table 1 (caption: "ELDR runtime overhead (Qwen3-30B-A3B, task, 8P16D, 60 req/s; median TTFT 69 ms).").

Total per-request serving overhead is 0.86 ms (1.2% of TTFT), dominated by the prefill-GPU reduce() scatter (0.48 ms) and D2H copy (0.21 ms). The scheduler fetch (7 µs) and routing decision (0.15 ms) are sub-percent — confirming the whole mechanism is a thin layer that does not perturb TTFT.

Baseline honesty (framework §7): baselines are Random, RR, JSQ, P2C, plus Domain (oracle domain-label locality — a stronger-than-realistic baseline since it uses ground-truth labels ELDR does not need). All six routers share the same prefill policy (PrefixHash) and decoder pool, so a cell isolates the decoder routing decision. Metrics are explicit: TPOT = time-per-output-token (median + p99), TTFT = time-to-first-token (median). vLLM version pinned (0.21.0rc1 / ROCm 7.2). This is a fair comparison; the one caveat is that Domain gets oracle labels, making ELDR's win over it a conservative measure of the signature's value.

Workload characterization (framework §6 — where it wins/loses):

Workload regime	ELDR	Best baseline	Why
Task (sharp domains, 1.41× skew)	−7.0–13.9% P50, −3.4–6.0% tail	Domain −6.8–9.7% P50	expert clusters align with labels; ELDR's finer K & τ-band beat static partition
Language (soft domains, 87.6% in top-4 langs)	−5.9–10.0% P50; tail mixed (−6.2% Qwen, +1.5% GPT-OSS, +0.2% Gemma mean)	load balancers	weaker locality (3–10%); Domain collapses (regresses tail up to 6.1%) as hot language block saturates
Large MoE + EP (235B, 2P8D TP4 EP4)	−2.7–4.3% P50, −0.6–2.0% tail	RR	needs per-decoder expert placement to avoid hot-expert EP-rank concentration

The honest loss surface: on language tail TPOT ELDR regresses 1.5% (GPT-OSS) and 0.2% (Gemma) on the mean (though all three reduce at the per-cell peak). And pure top-1 ($\tau=0$) — the maximal-locality extreme — actively regresses tail on 4/6 workloads.

6. 论证链 #

#	Step	Support (paper-internal)
1	MoE decode latency is governed by the union of distinct experts a batch activates, not token count.	§3.1 microbenchmark, Fig. 2: 16→128 experts = 4.7× latency; batch size ≈ flat.
2	This union is not random — expert usage is structured by request domain.	§3.2, Fig. 1: per-domain heatmaps show distinct over-activated expert subsets.
3	Same-domain colocation therefore shrinks the per-step union.	§3.4, Fig. 4: same-domain batches −17–21% (task) / −3–10% (language) experts.
4	The structure is visible before decode, because prefill uses the same gates.	§3.3, Fig. 3: prefill↔decode per-expert activation correlate 0.70–0.92.
5	A signature can capture it, if its distances rank pairs like true decode overlap.	§4.2.1 Eq. 1: $\rho$ = Spearman; §4.2.3 Fig. 7: count·idf beats continuous by >0.035.
6	Locality alone overloads popular domains; load must be reconciled.	§3.5.2, Fig. 5: WildChat top-2 langs = ~75% of traffic.
7	Split objectives by information: offline balanced K-means (aggregate) + online τ-band (instantaneous).	§4.3: Hungarian $\lceil N/K\rceil$ cap; band = $\{k: s_k \geq s^*-\tau\}$.
8	Prefix caching breaks signatures; block-granular co-indexed cache repairs them exactly.	§4.4, Fig. 10: $s_r=\sum_b \mathrm{sig}[b]$, exact across partial hits/evictions.
9	Net effect: lossless median+tail TPOT reduction, scaling with decoder pool.	§6.1 (−7.0–13.9%), §6.4 Table 2 (monotone 8.0→10.2%).

7. 实现 cross-reference #

Implementation status: described in §5 as ~2,000 LoC Python on vLLM 0.21.0rc1 / ROCm 7.2, but [实现未公开] — no public repository or file:line citations are provided in the source. The build has three components:

• Signature path — a prefill hook accumulates per-layer per-expert counts at KV-block granularity into an expert signature cache keyed by block id; at prefill completion the per-block signatures are summed and piggybacked onto the existing PD handoff (no new RPC).
• Per-decoder EP load balancing — borrows METRO (Yu et al., 2025): uses the calibration trace to estimate each cluster's per-expert decode activation, then greedily assigns the next most-active expert to the lightest-loaded EP rank per (decoder, layer). A single global EP layout would balance one cluster at the others' expense; per-decoder placement matches each decoder to its cluster while holding per-rank expert count fixed.
• Offline fit — one script produces the routing artifact (layer mask, IDF weights, $K$ Hungarian-balanced centroids in the masked/IDF/L2-normalized space) + expert placement file; re-run only on model/dataset/topology change, <10 s on CPU.

核心技术壁垒 (dedicated paragraph). The reproduction bottleneck is the $\rho$-based offline predictor (§4.2.1). Everything downstream — signature representation, IDF, greedy layer mask, even the $\tau$ default — is selected by a single scalar that is cheap to compute on 1,000 calibration prompts and provably tracks serving TPOT (Fig. 14 shows count·idf's higher $\rho$ maps to a further −3 pp, up to −14 pp, TPOT P50 vs. gate-prob). A naïve re-implementer who skips $\rho$ and instead sweeps the serving system per candidate representation would pay orders of magnitude more compute and likely still miss the layer-mask peak $N^$ (Fig. 8) that only $\rho$ surfaces. The insight is decoupling representation quality from the router* so the two can be optimized independently and offline.

关键实现细节 (easy-to-miss tricks).

1. Signature cache co-indexed with KV blocks, not per-request. A per-request signature cache only matches full prefix hits; ELDR stores one signature row per KV block so the block-sum $s_r=\sum_{b\in\mathcal{B}(r)}\mathrm{sig}[b]$ stays exact on partial hits, evictions, and reuse — inheriting the KV cache lifecycle with zero extra allocator/eviction/sync state.
2. Discrete count, not gate probability. The decode target $p_i$ counts top-$k$ selections; only the discrete top-$k$ actually loads an expert's weights. Continuous gate scores put mass on experts the top-$k$ never loads, and that mass cannot appear in $p_i$ — so the signature must be a discrete count to be a same-form estimator (Fig. 7, Fig. 14).

Software→hardware implication (framework §12 — conditional). Not triggered. ELDR is a pure software framework (request routing + prefix-cache-coherent bookkeeping + offline clustering); it introduces no persistent megakernel, cache-scope control, or interconnect-aware kernel. Its only hardware touch — the block-granular int8 signature cache accumulated in a single GPU pass and batched into the existing D2H copy — reuses primitives the paged KV cache already provides, so it argues for no new ISA/cache/interconnect feature.