DSpark bolts a lightweight sequential head (Markov or RNN) onto a parallel draft backbone (DFlash) to inject intra-block token dependency and stop suffix decay, then adds a calibrated confidence head + hardware-aware scheduler that prunes verification length per request to maximize throughput. +16–30% accepted length offline; 60–85% faster per-user generation vs MTP-1 in DeepSeek-V4 production.
Parallel drafters generate all $\gamma$ tokens in one forward pass, so drafting latency is nearly independent of block size — but this creates two failures when you actually try to exploit long blocks:
The latency model that frames everything is Eq. 1:
$$L = \frac{T_{\text{draft}} + T_{\text{verify}}}{\tau}$$
with three levers — lower $T_{\text{draft}}$ (draft faster), raise $\tau$ (draft better), or shrink effective $T_{\text{verify}}$ (verify smarter). Autoregressive drafters get high $\tau$ but pay $T_{\text{draft}} \propto \gamma$; parallel drafters collapse $T_{\text{draft}}$ but sacrifice $\tau$.
核心技术壁垒 (the single hardest-to-replicate insight): achieving load-aware, lossless verification-length scheduling. The scheduler must consume absolute-magnitude calibrated survival probabilities (not just rankings) to estimate $\tau$, yet the greedy search's early-stopping is exactly what preserves the non-anticipating property (admission of token $k$ must not depend on the realization of $x_k$). The genuinely subtle part — and the one that would trip a re-implementer — is that in production the early-stopping break is removed for jagged-hardware reasons, and losslessness is instead restored by an asynchronous "two-steps-prior" causal barrier. Getting this interaction right (calibration ↔ greedy monotonicity ↔ causality ↔ ZOS compatibility) is the moat.
No figures were extracted for this source; the decoding cycle (paper's Figure 1) and the two-module data path are reconstructed below as Mermaid diagrams.
Decoding cycle (prompt ABC → anchor D → draft EFGH → schedule → verify):
The parallel backbone does the expensive work once per block; the sequential head runs a cheap left-to-right loop to inject dependency; the confidence head scores each position; the scheduler truncates the block before it reaches the target model, so the target only verifies a load-appropriate prefix.
Two-module composition (where each equation lives):
Note the parameter coupling: the confidence head (Eq. 7) reuses the Markov embedding $W_1$ from the sequential head as a feature, so it conditions on the actually-sampled predecessor at essentially no extra parameter cost.
DSpark has one formal theoretical result (Appendix A losslessness counterexample) plus a set of analytical identities the design rests on. Notation table:
| Symbol | Meaning |
|---|---|
| $M_t, M_d$ | target / draft model |
| $\gamma$ | draft block size (proposal length) |
| $x_0$ | anchor (bonus) token from previous cycle |
| $U_k$ | base logit vector from parallel backbone at position $k$ |
| $B_k$ | prefix-dependent transition bias from sequential head |
| $p^d_k, p^t_k$ | draft / target distribution at position $k$ |
| $\tau$ | accepted tokens per cycle |
| $c_k \in (0,1)$ | predicted per-position survival prob (given prefix accepted) |
| $c^*_k$ | analytical (ground-truth) per-step acceptance rate |
| $a_{r,j}=\prod_{i\le j}c_{r,i}$ | prefix survival probability of request $r$ at position $j$ |
| $\text{SPS}(B)$ | profiled engine throughput (steps/sec) at batch $B$ |
| $\Theta = \tau\cdot\text{SPS}(B)$ | expected system token throughput |
方程物理意义 (key equations):
Monotonicity / greedy-optimality proposition (§3.2.2, informal). Statement: because $a_{r,j}\le a_{r,j-1}$ (survival probability is non-increasing in $j$), the marginal gain of extending request $r$ from $j-1$ to $j$ is exactly $a_{r,j}$. Therefore, for a fixed batch size $B$, the optimal length allocation is obtained by greedily admitting the globally-highest-$a_{r,j}$ tokens; sorting descending by $a_{r,j}$ automatically respects intra-block prefix ordering. Assumption: stepwise early-stopping yields the global-max $\Theta$ iff $\Theta$ is unimodal in $B$, which requires a smoothly decaying SPS curve. When it breaks: real hardware SPS is jagged/step-wise (tensor-core tile quantization), so the unimodality assumption fails in production — addressed in §7 below.
Losslessness counterexample (Appendix A, reproduced). Setup: $R=1$, $\gamma=2$, $a_1=0.8$, capacity $\text{SPS}(1)=1.0,\ \text{SPS}(2)=0.5,\ \text{SPS}(3)=0.45$.
$$\Theta_0 = 1\cdot 1.0 = 1.0, \qquad \Theta_1 = (1+0.8)\cdot 0.5 = 0.9.$$
Without early-stopping the scheduler evaluates $\Theta_2$, but $c_2$ (and thus $a_2=a_1 c_2$) depends on the realization of $x_1$: if $c_2=0.9$ then $\Theta_2=(1+0.8+0.72)\cdot0.45=1.134$ (returns $\ell=2$, admits $x_1$); if $c_2=0$ then $\Theta_2=0.81$ and the max stays $\Theta_0$ (returns $\ell=0$, rejects $x_1$). So admission of $x_1$ depends on $x_1$ — selection bias. Made explicit with vocab $\{A,B\}$, $p^t=(0.7,0.3)$, $p^d=(0.5,0.5)$: acceptance $\sum_x\min(p^t,p^d)=0.8$ matches $a_1$. The retrospective scheduler yields
$$\Pr(Y=A) = 0.5\cdot 1 + 0.5\cdot 0.7 = 0.85 \ne p^t(A)=0.7,$$
proving the global-search scheduler is not lossless. Early-stopping fixes it: since $\Theta_1<\Theta_0$, the scheduler halts at $\ell=0$ before touching any $x_1$-dependent quantity, restoring the non-anticipating property.
Six minimum checks:
No figures were extracted; the load-bearing tables/plots are reproduced numerically below.
Table 1 — accepted length $\tau$ per round (scheduler disabled, fixed block). DSpark is bold/best in every cell.
| Target | Drafter | GSM8K | MATH | AIME25 | MBPP | HumanEval | LCB | MT-Bench | Alpaca | Arena-Hard |
|---|---|---|---|---|---|---|---|---|---|---|
| Qwen3-4B | Eagle3 | 5.14 | 4.62 | 3.92 | 3.69 | 4.16 | 3.77 | 2.39 | 2.26 | 2.55 |
| Qwen3-4B | DFlash | 5.40 | 4.85 | 4.15 | 4.40 | 4.74 | 4.18 | 3.07 | 2.96 | 2.83 |
| Qwen3-4B | DSpark | 6.11 | 5.70 | 4.89 | 5.13 | 5.38 | 4.86 | 3.64 | 3.54 | 3.29 |
| Qwen3-8B | Eagle3 | 5.30 | 4.77 | 3.91 | 3.96 | 4.33 | 4.17 | 2.66 | 2.54 | 2.54 |
| Qwen3-8B | DFlash | 5.33 | 4.91 | 4.07 | 4.36 | 4.64 | 4.39 | 3.11 | 2.98 | 2.81 |
| Qwen3-8B | DSpark | 6.17 | 5.78 | 5.01 | 5.16 | 5.52 | 5.17 | 3.72 | 3.58 | 3.21 |
| Qwen3-14B | Eagle3 | 5.24 | 4.60 | 3.71 | 3.81 | 4.14 | 4.01 | 2.62 | 2.47 | 2.48 |
| Qwen3-14B | DFlash | 5.41 | 4.84 | 3.98 | 4.44 | 4.59 | 4.33 | 3.10 | 2.94 | 2.72 |
| Qwen3-14B | DSpark | 6.21 | 5.74 | 4.94 | 5.26 | 5.43 | 5.02 | 3.70 | 3.58 | 3.13 |
| Gemma4-12B | Eagle3 | 5.87 | 5.46 | 4.83 | 4.72 | 5.37 | 4.16 | 3.19 | 3.06 | 2.72 |
| Gemma4-12B | DFlash | 5.45 | 5.04 | 4.22 | 4.39 | 4.95 | 3.70 | 2.98 | 2.84 | 2.59 |
| Gemma4-12B | DSpark | 6.05 | 5.78 | 5.12 | 5.11 | 5.64 | 4.51 | 3.49 | 3.35 | 2.92 |
The table also exposes a strong domain effect (Qwen3-4B: ~5.6 math, ~5.1 code, ~3.5 chat). This inherent variance is what makes a static verification length wasteful and directly motivates the scheduler.
Position-wise conditional acceptance (Fig. 2, numeric). With the penalty of earlier rejections removed, Eagle3 is flat/rising with position while DFlash decays:
Depth & length ablations (Fig. 3/4, numeric).
Confidence-head validation (Fig. 5/6, numeric).
Production (Fig. 7/8, numeric).
| # | Step | Paper-internal support |
|---|---|---|
| 1 | Per-token latency is $L=(T_{\text{draft}}+T_{\text{verify}})/\tau$, giving three optimization levers. | Eq. 1 (§2.1) |
| 2 | Parallel drafters collapse $T_{\text{draft}}$ but lose $\tau$ because independent positions cause multi-modal collision and suffix decay. | §3.1 para 1; Fig. 2 tail decay (§4.3.1) |
| 3 | Adding a locally-normalized sequential bias $B_k$ (Eq. 4–6) restores intra-block dependency while keeping exact per-token probabilities and near-parallel speed. | Eq. 4–6 (§3.1); latency +0.2–1.3% (Fig. 4) |
| 4 | This yields both high position-1 acceptance and a flat tail, so accepted length beats both AR and parallel baselines. | Fig. 2 + Table 1 (§4.2–4.3.1) |
| 5 | But longer accepted blocks don't help throughput unless verification is load-aware; a confidence head (Eq. 7) supervised by the TV-distance acceptance rate (Eq. 8) predicts survival probability. | §3.2.1; Fig. 5 pruning |
| 6 | The scheduler must consume absolute calibrated probabilities to estimate $\Theta=\tau\cdot\text{SPS}(B)$; STS calibrates cumulative products to ~1% ECE without disturbing rankings. | §3.2.1; Fig. 6 |
| 7 | Verification-length selection is then a throughput-max problem solvable greedily by survival-probability monotonicity, with early-stopping guaranteeing losslessness. | Alg. 1 (§3.2.2); Appendix A counterexample |
| 8 | In production, jagged SPS + ZOS force removing early-stopping; an asynchronous two-steps-prior "causal barrier" restores losslessness, shifting the DeepSeek-V4 Pareto frontier. | §5.2; Fig. 7/8 (§5.4) |
The paper open-sources DSpark checkpoints (DeepSeek-V4-Flash/Pro preview) and DeepSpec, an algorithm-driven training repository including Eagle3, DFlash, and DSpark (github.com/deepseek-ai/DeepSpec). Specific file:line citations to the released training/inference code are not enumerated in the L1 source. [实现未公开 — 代码仓库已开源但 L1 未记录具体 file:line]
核心技术壁垒 (dedicated paragraph). The load-bearing, hard-to-replicate insight is the interaction between calibration, greedy monotonicity, and causality under a real serving pipeline. The scheduler needs absolute-magnitude survival probabilities (Fig. 6 shows raw scores are 3–8% overconfident and would distort $\tau$ estimation), so STS calibration is a prerequisite, not a nicety. The greedy admission is only optimal because $a_{r,j}$ is monotone; and losslessness hinges on the non-anticipating property, which the offline algorithm secures with an early-stopping break (Appendix A shows removing it yields output $(0.85,0.15)\ne(0.7,0.3)$). Production then removes that break to escape jagged-SPS local minima — and re-earns losslessness only because the asynchronous scheduler evaluates predictions from two steps prior, forming a temporal causal barrier that isolates the admission decision from the current token's realization (§5.2). Reproducing DSpark's headline production numbers requires getting all four of these to hold simultaneously alongside ZOS/CUDA-graph compatibility.
关键实现细节 (easy-to-miss tricks):