SPECTRE: Hybrid Ordinary-Parallel Speculative Serving — L2 #

1. TL;DR #

SPECTRE reuses idle tail-model services as remote speculative drafters for busy large models, and adaptively switches between parallel (draft overlaps verify) and ordinary (draft waits) modes using a throughput-derived rollback threshold $r^*$. Up to 2.28× over AR, +66% over the best SD baseline.

2. Q1 / Q2 / Q3 #

Q1 — 痛点. Multi-model serving clouds have long-tailed demand: a few large models are saturated while many small tail models sit idle. Speculative decoding (SD) can borrow that idle small-model capacity as a remote drafter, but naive parallel SD — where drafting overlaps target verification — is not always a win. When a pre-generated draft is invalidated by the verifier (a rollback), the overlapped work is wasted and the effective accepted length collapses to 1. Under high concurrency or a TP-heavy target (verification fast relative to draft), rollbacks dominate and parallel SD can fall below autoregressive decoding (PEARL drops to 0.36× at bs=128).

Q2 — 方法. Three coupled mechanisms:

1. Hybrid ordinary-parallel execution: per-round, compare the observed rollback ratio $r_n$ against a closed-form threshold $r^$ derived from a throughput model; run parallel when $r_n \le r^$, else fall back to ordinary (target waits for repaired drafts). This is the piece that keeps accepted length near the ordinary-SD ceiling while still harvesting overlap when it pays.
2. Speculative priority scheduling on the shared draft server: non-preemptive priority for speculative requests, with a fairness valve every $K$ steps so background user traffic is not starved.
3. Draft-side context compression (StreamingLLM-style prefix+suffix keep, ratio $p$): shrinks draft prefill/decode latency so drafting stays off the critical path when the target is TP-accelerated.

核心技术壁垒: the closed-form switching threshold $r^$ that decides, per round and per batch*, whether overlap is net-positive — turning "parallel vs. ordinary SD" from a static design choice into a runtime-adaptive controller (see §7).

Q3 — 结果. Up to 2.28× over AR and up to +66% over the strongest SD baseline (DS-32B bs=128 LongBench). Under TP8 (Qwen3-235B-A22B), +12–19% over the best competitor where PEARL goes sub-AR. Shared draft traffic costs the target <2% throughput up to 8 QPS. System-level benefit (\$/1000s/GPU) up to 1.81× AR because the draft GPU also serves paying traffic.

3. 架构 / 方法图 #

SPECTRE is a disaggregated serving framework: a target server (large model, possibly TP=8) and one or more draft servers (small tail models, TP=1) that also serve their own native traffic. Communication is a ZMQ router/dealer backend (C++ extension, local IPC on-node, TCP cross-node). It owns the scheduling/coordination axis; it delegates tensor/pipeline parallelism to the underlying SGLang runtime.



Paper's Figure 1. A large-model (target, blue) service pulls speculative drafts over ZMQ from an underutilized tail-model (draft, orange) service; the draft server keeps serving its own native requests under speculative-priority scheduling while answering target draft queries. The key architectural notice: the drafter is not a dedicated sidecar — it is a live multi-tenant service, which is what makes the system-benefit story work.



Paper's Figure 2. Timeline contrast: ordinary SD is fully serialized (draft → verify → draft); parallel SD overlaps draft generation with target verification; SPECTRE adaptively switches. Notice that parallel mode's per-round latency shrinks to $T_T$ (verification only) — the entire benefit case — but only when the pre-generated draft survives verification.



Paper's Figure 3. Draft model $M_D$ generates candidates while serving normal requests; target $M_T$ verifies via rejection sampling. When the rollback proportion is high the system switches to ordinary coordination (target waits for repaired draft tokens); otherwise execution proceeds overlapped without interruption. This is the runtime realization of the §3.1 mode-switch rule.

The request/task lifecycle: request arrives at target → target assembles a verification candidate $\mathbf{y}_n^{(i)}$ per request → merges the batch $\mathbf{Y}_n$ → verifies (tree degenerates to a chain of length $\gamma$ in the common case) → commits verified tokens → sends bonus token + round id to draft server → draft reconciles state and generates the next $\gamma$-token continuation. The scheduler on the draft side is FCFS-with-speculative-priority (non-preemptive, fairness every $K$ steps). The KV/state manager is per-request persistent speculative state $\mathcal{S}_n^{(i)}=(\mathbf{h},\mathbf{d},\mathbf{k})$ with rollback keyed on the first divergence point.

4. 作者证明 #

The framework's core is a throughput model (§2 + Appendix C, Eqs. 39–50) comparing ordinary and parallel modes and solving for the crossover rollback ratio.

Notation table.

Symbol	Meaning
$B$	batch size
$L$	accepted length when the draft is valid
$\gamma$	speculative tokens verified per round
$T_D$	latency of one draft step
$T_T$	latency of one target verification
$r$	rollback ratio (fraction of requests whose pre-gen draft is invalidated)
$r^*$	crossover threshold: ordinary beats parallel above it

方程物理意义.

Ordinary throughput — one verification plus $\gamma-1$ fresh draft steps per round, all serialized:

$$\mathrm{Thr}_{\mathrm{ord}} = \frac{BL}{T_T + (\gamma-1)T_D}$$

Parallel effective length — rollback requests salvage only the bonus token (1), others keep $L$:

$$L_{\mathrm{par}} = r\cdot 1 + (1-r)\cdot L$$

Parallel throughput — drafting is hidden, so round latency is just $T_T$:

$$\mathrm{Thr}_{\mathrm{par}} = \frac{B(r + (1-r)L)}{T_T}$$

Setting $\mathrm{Thr}_{\mathrm{ord}} > \mathrm{Thr}_{\mathrm{par}}$ and solving for $r$ gives the threshold:

$$r^{*} = \frac{(\gamma-1)L\,T_D}{(T_T + (\gamma-1)T_D)(L-1)}$$

6 checks.

1. Denominator of $\mathrm{Thr}_{\mathrm{par}}$ is $T_T$, not $T_T+(\gamma-1)T_D$ — correct: in parallel mode drafting overlaps verification, so it is excluded from the critical path. This exclusion is exactly what parallel mode buys and why $T_D$ vanishes from the parallel denominator.
2. Why "1" for rollback requests in $L_{\mathrm{par}}$ — a rolled-back draft is invalid; only the target-verified bonus token survives, contributing a single guaranteed token. Consistent with Eq. 12's padded candidate $[b, \varnothing,\dots]$.
3. Boundary/monotonicity: $r^$ increases in $T_D$ and decreases in $T_T$ — larger draft latency makes overlap more valuable (tolerate more rollback), while a fast target ($T_T\downarrow$, the TP8 case) shrinks $r^$ toward 0, i.e., parallel mode is almost never worth it. This matches the §4.3 finding that pure-parallel PEARL goes sub-AR under TP8.
4. $L>1$ requirement: division by $(L-1)$ demands $L>1$; if $L=1$ (no acceptance beyond bonus) SD has no benefit at all, so the regime is degenerate — the paper explicitly assumes $L>1$ (Eq. 50).
5. First-order sanity on the operating assumption $\gamma T_D < T_T$ (Eq. 39): with $T_T$ large relative to $\gamma T_D$, $r^$ approaches $\frac{(\gamma-1)L T_D}{T_T (L-1)}$, a small positive number — so ordinary is preferred unless rollbacks are rare. As $T_T \to (\gamma-1)T_D$ the threshold rises toward $\frac{L}{L-1}>1$, meaning parallel is always* preferred (any $r\le 1$ passes). Both limits are physically sensible.
6. Interior vs. boundary optimum: the decision is a threshold test on a single scalar $r_n$; the "optimum" is a boundary switch, not an interior calculus optimum. The controller therefore only needs to measure $r_n$ each round — no per-round optimization. This is why it is cheap enough to run online.

Caveat the paper does not close: $r^*$ treats the whole batch with one rollback ratio; per-request rollback heterogeneity is not modeled.

5. 实验与数据 #



Paper's Table 1. Throughput (Tok/s, speedup vs AR) for Qwen3-32B and DS-R1-32B targets at bs∈{32,64,128}. SPECTRE is top in every reported TP1 cell; the load-bearing wins are on LongBench (long context), e.g. Qwen3-32B bs=64 reaches 2.28× AR and DS-32B bs=128 LongBench is +66.4% over the best baseline. Note the failure modes it beats: EAGLE3 drops to 0.65–0.81× on LongBench, and PEARL collapses to 0.36× at bs=128 — evidence that naive parallel SD is fragile exactly where SPECTRE's ordinary fallback engages.



Paper's Figure 4. bs=128 with Qwen3-235B-A22B (TP8) target, Qwen3-0.6B (TP1) draft, on LongBench-v2 / MRCR / ShareGPT. SPECTRE leads by 19% / 12% / 14% over the strongest competitor. The critical observation: PEARL falls below AR here — the TP8 target verifies so fast that the single-GPU draft becomes the bottleneck, exactly the $T_T$-small regime where §4's model predicts $r^*\to 0$ and pure parallel loses.



Paper's Table 2. System benefit including draft-side revenue. SPECTRE is highest on all pairs (up to 1.81× AR on DS-R1). The damning row: PEARL on Qwen3-235B is 0.16 vs AR's 0.38 — a dedicated, non-shared draft GPU is a net economic loss. SPECTRE wins here precisely because its draft server keeps earning on native traffic.



Paper's Table 3. Accepted length averaged over batch sizes. Standalone (ordinary SD, no parallel interference) is the ceiling (3.13–3.77); SPECTRE stays close (2.59–3.61) while pure-parallel EAGLE3/PEARL are much shorter (1.66–2.44). This is the mechanistic proof that the $r^*$ switch preserves acceptance — SPECTRE trades a little length for a lot of overlap, whereas PEARL loses both.

Additional (Tables 4–5, not embedded here): bs∈{1,16} still SPECTRE-best (2.36–3.12× at bs=1), and temperature-0.6 sampling preserves the gains, growing with batch size.

6. 论证链 #

Step	Claim	Support (paper-internal)
1	Multi-model clouds leave tail-model GPUs idle; those can serve as remote drafters.	§1 long-tail framing; Fig. 1 topology
2	Parallel SD overlaps draft+verify but rollbacks cut effective length to 1, so overlap is only sometimes net-positive.	§2 Eqs. 1,3,4; Fig. 2 timeline
3	A closed-form threshold $r^*$ (Eq. 50) marks where ordinary beats parallel; switch modes per round on measured $r_n$ (Eq. 5).	§2 + Appendix C derivation
4	Switching keeps accepted length near the ordinary-SD ceiling.	§4.5 Table 3 (SPECTRE 2.59–3.61 vs Standalone 3.13–3.77)
5	Draft latency must stay below $T_T$; priority scheduling + context compression enforce this even under shared traffic and TP8.	§3.2 Eqs. 16–17; §4.3 Fig. 4; §4.4 Fig. 5 (<2% target loss)
6	Net effect is highest throughput and highest \$/GPU across all regimes tested.	§4.2 Table 1; §4.4 Table 2

7. 实现 cross-reference #

Implementation is in SGLang, PR available: sgl-project/sglang#22272 (evaluated on SGLang v0.5.7). Concrete file:line citations are not in the L1; treat below as component-level pointers into that PR. [实现未公开 at file:line granularity — see PR #22272]

核心技术壁垒 (elaboration). The single hardest-to-replicate insight is the runtime rollback-threshold controller. Anyone can implement parallel SD or ordinary SD; the contribution is measuring $r_n$ each round and comparing to the analytically-derived $r^*$ (Eq. 50), which itself depends on live $T_D$, $T_T$, $\gamma$, $L$. Reproducing it requires (a) instrumenting per-round rollback counts on the target, (b) a stable estimate of $L$ and the two latencies under batching, and (c) a mode switch that can re-assemble the verification batch differently (ordinary: fresh repaired drafts, Eqs. 6–10; parallel: padded rollback candidates, Eqs. 12–15) without stalling. Getting acceptance length to sit near Standalone (Table 3) is the observable evidence the controller is tuned correctly.

关键实现细节 (easy-to-miss tricks).

1. Round-and-request-id validation of async replies (Eq. 24): a draft reply is accepted only if both $(i,n)$ match the current request/round — without this, stale or reordered ZMQ replies silently corrupt generation state. There is also a circuit breaker (Eqs. 26–27): after $C_{\max}$ missed replies, remote speculation is disabled for $H$ rounds.
2. Context compression is only prefix+suffix keep (Eq. 17, $p=0.1$ ⇒ first 5% + last 5%): dropping 90% of the draft-side context still works because the throughput saving from lower draft latency outweighs the acceptance-length hit — but this only holds when draft latency is on the critical path (TP8 target). At TP1 the paper disables compression.

§10 Deployment context. Both stages (prefill+decode) run on the target; the draft server is a co-tenant serving system, not a sidecar. Best regime: high-concurrency (bs≥32) long-context on a TP-heavy target where a small idle model already exists in the fleet. Integration cost: a scheduler + executor patch to SGLang plus a C++ ZMQ transport — a fork/patch, not a config flag. Migration for a vLLM/SGLang shop: adopt the SGLang PR and stand up (or repurpose) a small-model endpoint as the drafter.

(§12 software→hardware reverse implication: not triggered — SPECTRE is a scheduling/coordination framework, not hardware-proximal.)