Mode A · target category = framework. ELDR adds expert locality as a second
decode-routing axis on top of a PD-disaggregated MoE stack, losslessly cutting
median TPOT 5.9–13.9%.
The 8 related entities do not form one tight cluster; ELDR sits at the
intersection of three lineages, and its novelty is precisely that it borrows
mechanisms from all three without belonging to any.
A — PD-disaggregation + KV-centric serving (direct ancestors).
"routing is the central serving decision" framing, and a distributed KVCache
pool with prefix reuse [2407.00079]. ELDR literally
builds on top of a Mooncake-style stack — it adopts PrefixHash prefill
routing (Mooncake's cache-aware prefill lineage) and adds a decode-side signal
Mooncake never modeled.
block-granular signature cache is co-indexed with the paged KV cache and
inherits its block lifecycle — a direct structural dependency on PagedAttention's
logical→physical block table [2309.06180].
B — decode-phase memory-bandwidth optimization (mechanism cousins).
bottleneck (memory-bandwidth-bound attention/weight loading) but by
offloading KV/attention to CPU [2403.11421]
[2411.01142]. They move where computation happens; ELDR
moves which requests are colocated. Orthogonal knobs on one bottleneck.
thesis — 90% of latency is queueing, not execution [2305.05920].
ELDR's analogue: the win is in routing composition, not the decode kernel.
C — MoE substrate & training-side placement (the workload & a distant echo).
pointedly, names the exact failure mode ELDR exploits: expert load imbalance
from domain shift, mitigated only by redundant expert deployment
[2412.19437]. ELDR is a decode-routing answer to a
problem DeepSeek-V3 flagged at the model level.
not serving — but both pioneer the disaggregate-by-role / place-by-property
paradigm (training vs. generation runtimes on separate device groups)
ELDR is the same design instinct pushed one level finer: place requests (not
models) by an intrinsic property (expert affinity, not train/infer role).
New (no peer has it):
here treats decode workers as interchangeable once load is balanced. Mooncake
routes by KV cache affinity and SLO [2407.00079]; vLLM's
scheduler is FCFS load [2309.06180]; FastServe by predicted
job time [2305.05920]. ELDR is the first to make *which
experts a colocated batch's union activates* the routing objective, backed by
the 4.7× latency-vs-active-experts microbenchmark [2607.00466].
correlation [2607.00466] is a genuinely novel exploited
signal; no peer reads model-internal gate activations for routing.
ELDR contrasts against (dropping/piggybacking/sharing experts), ELDR changes
only which worker serves a request — outputs bit-identical to top-k gating
[2607.00466]. This is a stronger correctness guarantee than
FP8 training's "<0.25% loss error" [2412.19437] or any
quantized-serving trade-off.
Incremental / borrowed:
invented [2407.00079].
discipline; the signature cache is "just another store co-indexed at the same
block id" [2309.06180].
scope), the same "balance activated experts, not tokens" idea DeepSeek-V3's
redundant-expert deployment prefigures [2412.19437].
offline auto-mapping + runtime execution split [2409.19256]
— aggregate structure decided offline, instantaneous state handled online.
Contradictory / tension:
(active-expert union dominates) [2607.00466]. FastDecode and NEO
are built on the opposite emphasis: batch size is the throughput lever, so
aggressively grow it by offloading KV [2403.11421]
矛盾根源: different metric + regime. ELDR optimizes **latency (TPOT) at a
fixed batch* in the memory-bound union* regime of fine-grained MoE
(E=128, many distinct experts); FastDecode/NEO optimize throughput for
dense/coarse models where one weight load amortizes over the batch and batch
size directly scales GPU utilization. Both are correct in their own regime —
ELDR's "batch size ≈ neutral" holds only because expert fragmentation
destroys the weight reuse FastDecode/NEO's batching relies on. This is the
single most important cross-paper distinction in the cluster.
4.7× vs. active experts result is measured on a single MoE layer, one MI300X,
at batch 64 [2607.00466]. At very large batches the token
count per expert eventually re-densifies each expert's GEMM and batch size
does re-enter as a compute term. ELDR never sweeps batch high enough to show
where the crossover is — the same "regime not shown" weakness NEO admits about
its own iteration model [2411.01142].
only 3–10% on language vs. 17–21% on task [2607.00466], and
ELDR regresses tail TPOT on 2 of 3 language models on the mean. An adversary
can argue the headline "5.9–13.9%" leans on the sharp-domain task workload;
the realistic MaaS traffic (WildChat-like, heavily language-skewed) is the
weaker case — precisely the traffic Mooncake actually serves in production
but Domain is given oracle ground-truth labels [2607.00466].
The honest reading: ELDR's signature is competitive with an oracle — strong
— but the paper frames beating an oracle-fed baseline as the headline win,
which overstates the margin over a realistic label-free locality scheme.
the gate decision; ELDR still changes which requests batch together, and any
numerically-sensitive kernel fusion or reduction order across a differently-
composed batch could perturb low bits. The paper asserts exactness by
construction [2607.00466] but does not report an output-diff
audit at scale — a claim worth verifying, not assuming.
fit on a 1,000-prompt calibration split. Under genuine workload drift (new
domains, seasonal language shift) the signature space geometry moves, and there
is no online drift detector — the same static-configuration criticism leveled at
FlexRLHF's pre-training-fixed placement [2312.11819].
This is ELDR's strongest ecosystem card. FastDecode, NEO, and Mooncake each
require adopting a new engine or CPU cluster [2403.11421]
[2411.01142] [2407.00079]. ELDR is
~2,000 LoC of Python over an unmodified vLLM PD stack
[2607.00466] — model, kernels, batching, engines
all untouched. Its migration cost is near-zero relative to every serving peer
in scope.
(a) prefix caching (the block-granular signature cache), (b) intra-worker EP
balancing (METRO / DeepSeek-V3-style redundant experts run inside each
decoder) [2412.19437], and (c) CPU offloading in
principle (NEO/FastDecode could sit under an ELDR decoder). It occupies the
one empty layer — inter-worker decode routing by model-internal state —
that the disaggregation lineage left open.
GPT-OSS) made the E=128 "many distinct experts" regime the norm
[2412.19437]; the technique would have been nearly
worthless in the coarse-expert era where the union rarely exceeds the batch's
natural expert count. ELDR is a consequence of the frontier-open-MoE trend,
not independent of it.
[实现未公开] — no public repoor file:line, unlike vLLM (de-facto standard, fully open
[2309.06180]), Mooncake (open trace + code
[2407.00079]), or HybridFlow/veRL (active community
[2409.19256]). Reproducibility currently rests on the
paper text alone; the $\rho$-based offline predictor is the described-but-not-
shipped crux [2607.00466].
Hybrid / adaptive directions surfaced by triangulating the cluster:
expert union; NEO offloads the attention half to CPU
[2411.01142]. A decoder whose expert union is small has
spare HBM for a larger batch — exactly the batch NEO wants to feed its CPU
pipeline. A joint scheduler that routes by expert-locality and sizes the
CPU-offload batch per-decoder is unexplored, and directly reconciles the §2
batch-size contradiction.
handoff for decode routing. The same signature could steer prefill worker
choice too, unifying Mooncake's KV-affinity prefill routing
[2407.00079] with expert-affinity into a single
prefill+decode co-placement objective.
streaming balanced-K-means that tracks distribution shift — mirroring the
move HybridFlow/FlexRLHF have not made either (both stay static)
so it is an open direction for the whole placement-framework family, not just
ELDR.
auxiliary-loss-free bias balances experts at train time
[2412.19437]; ELDR reveals a serving-time domain
structure. Feeding observed inference-time expert-locality clusters back into
redundant-expert placement decisions closes a train↔serve loop neither side
currently exploits.
where MoE decode transitions from union-bound (ELDR's regime) to batch-bound
(FastDecode/NEO's regime). Mapping that curve for fine-grained MoE would tell a
deployer when to reach for locality routing vs. offloading — the missing
decision boundary between the two sub-clusters.