World Action Models are Zero-shot Policies (DreamZero) — L2 #

1. TL;DR #

DreamZero is a 14B autoregressive video-diffusion backbone fine-tuned to jointly denoise future frames and actions. By treating action learning as inverse dynamics conditioned on a predicted visual future, it beats SOTA VLAs 2× on unseen-task/env generalization, transfers cross-embodiment from video-only data, and — via a 38× inference-optimization stack — runs closed-loop at 7Hz.

---

2. Q1 / Q2 / Q3 #

Q1 — 痛点 (Problem formulation) #

VLAs inherit semantic priors ("what to do") from web-scale VLM pretraining but lack physical priors ("how to execute") — they fail on unseen motions like "untie the shoelace" because a fixed library of episode-level language-conditioned motion primitives cannot cover the space of possible physical interactions. The paper reframes policy learning as inverse dynamics on top of a learned world model. The objective factorizes the joint distribution of video and action:

$$\pi_{0}(\mathbf{o}_{l:l+H},\mathbf{a}_{l:l+H}\mid\mathbf{o}_{0:l},\mathbf{c},\mathbf{q}_{l})=\pi_{0}(\mathbf{o}_{l:l+H}\mid\mathbf{o}_{0:l},\mathbf{c},\mathbf{q}_{l})\,\pi_{0}(\mathbf{a}_{l:l+H}\mid\mathbf{o}_{0:l+H},\mathbf{q}_{l})$$

The IDM term is conditioned on observations including the predicted future $\mathbf{o}_{0:l+H}$ — actions are read off from an imagined visual plan rather than mapped from the current state. Assumptions: the trajectory is treated as a chunked autoregressive sequence (a Markov-like conditioning on clean prior chunks under teacher forcing); one training step consumes a sampled trajectory split into $M$ chunks and produces a flow-matching velocity-regression gradient over both modalities.

Q2 — 方法 (Method core) #

A single end-to-end DiT jointly denoises a video latent chunk and its aligned action chunk with a shared flow-matching objective, trained autoregressively with teacher forcing (attend to clean previous chunks). At inference, ground-truth observations replace predicted frames in the KV cache after each chunk executes — this is the mechanism that kills the compounding-error problem of pure autoregressive video generation.

The single novel mechanism vs prior WAMs, as a before/after diff:

	Prior WAMs	DreamZero
Architecture	bidirectional, fixed-length (subsamples video, distorts FPS)	autoregressive chunks, native FPS, KV-cached
Timestep coupling	separate video/action denoising schedules	shared $t_k$ in base model; decoupled only in Flash
Closed-loop error	accumulates from predicted frames	GT frames injected into KV cache each chunk
Data philosophy	repetitive per-task demos	breadth-first heterogeneous, non-repetitive

核心技术壁垒 (the single hardest-to-replicate insight): the decoupled noise schedule of DreamZero-Flash. The observation that at few-step inference video tokens are still noisy, so the model must be trained to predict clean actions from deliberately noisy video conditioning — implemented by biasing $t_k^{\text{video}}$ toward high-noise via a $\text{Beta}(\alpha,\beta),\ \alpha>\beta$ prior while keeping action timesteps uniform. This closes the train/test mismatch that otherwise collapses 1-step action quality (52% → 74%). Combined with the GT-KV-injection trick, this is the load-bearing recipe detail; see §7.

Q3 — 结果 (Results) #

• Seen-task (unseen env/objects) on AgiBot G1: 62.2% avg task progress vs best pretrained VLA 27.4% (>2×); from-scratch VLAs ≈ 0%.
• Unseen-task zero-shot on AgiBot: 39.5% vs 16.3%; on DROID-Franka 49% task progress / 22.5% success vs ≤33% / ≤12.5%.
• Cross-embodiment (video-only, 10–20 min): 38.3% → 55.4% robot-to-robot, 54.3% human-to-robot.
• Few-shot new embodiment (30 min play data on YAM): retains language following, generalizes to novel objects.
• Inference: 38× speedup (5.7s → 150ms), enabling 7Hz closed-loop control on 2× GB200.

---

3. 架构 / 方法图 #

Paper's Figure 4, verbatim (caption: "Model Architecture of DreamZero. The model takes three inputs: visual context (encoded via a VAE), language instructions (via a text encoder), and proprioceptive state (via a state encoder). These are processed by an autoregressive DiT backbone using flow matching, which jointly predicts future video frames and actions through separate decoders. During training (left)... During inference (right), predictions are executed asynchronously in the real world, and ground-truth observations are fed back into the KV cache to prevent error accumulation.").

This is the load-bearing architecture figure: note that only three lightweight modules (state encoder, action encoder, action decoder) are added to the frozen video backbone, and that the training/inference asymmetry (predicted frames on the left, GT-replaced frames on the right) is the diagram of the compounding-error fix. The reader should notice the video modality is autoregressive but action prediction is not fed back — errors do not propagate through the action stream.

Paper's Figure 14, verbatim (caption: "Attention strategy of DreamZero. (a) QKV Self-Attention mask for training... the model to predict velocities of next frames (Z1, Z2, Z3) and actions (Y1, Y2, Y3). (b) During inference, we compute the KV-cache of conditional frames... Note that C0, C1, C2 during inference is replaced with the GT observations.").

This figure operationalizes Eq. 3's teacher forcing: the current noisy chunk attends only to clean context of previous chunks, and at inference those context slots hold ground-truth observations. It shows precisely how "action $Y_3$ attends to $C_0, C_1, C_2$" — the visual-history conditioning that makes DreamZero a stateful policy.

Paper's Figure 2, verbatim (caption: "Joint Video and Action Prediction. DreamZero jointly generates video and action. We observe that the predicted actions closely align with the generated video. The examples are from totally unseen tasks.").

This motivates the whole design: the predicted action overlay tracks the imagined video, visually confirming that the IDM term reads actions off the visual plan even on tasks absent from training.

---

4. 作者证明 #

无形式化作者证明 — 仅实证 (no convergence theorem or bound). DreamZero is a training-recipe / architecture paper; it provides no formal convergence, variance, or sample-complexity guarantee. What would have been desirable: a bound relating video-prediction error to downstream action error (the paper's central empirical claim — "policy performance is fundamentally tied to video generation quality" — is exactly the kind of statement a Lipschitz-style error-propagation lemma could formalize). The formal content is instead the objective definition and loss decomposition, verified below.

Notation table

Symbol	Meaning
$\mathbf{o}_{l:l+H}$	future video frames over horizon $H$
$\mathbf{a}_{l:l+H}$	future actions over horizon $H$
$\mathbf{o}_{0:l}$	observation history through index $l$
$\mathbf{c},\ \mathbf{q}_l$	language instruction; proprioceptive state
$\mathbf{z}_{t_k}^{k},\ \mathbf{a}_{t_k}^{k}$	noisy video latent / action for chunk $k$ at timestep $t_k$
$\mathcal{C}_k=\{(\mathbf{z}_1^j,\mathbf{a}_1^j)\}_{j=1}^{k-1}$	clean context of previous chunks (teacher forcing)
$\mathbf{u}_\theta$	joint video-action DiT predicting velocity
$\mathbf{v}^k$	target velocity $[\mathbf{z}_1^k,\mathbf{a}_1^k]-[\mathbf{z}_0^k,\mathbf{a}_0^k]$
$w(t_k)$	per-timestep loss weight
$\eta\sim\text{Beta}(\alpha,\beta)$	Flash video-noise bias variable, $\alpha>\beta$

方程物理意义

• Eq. 1 (factorization): the joint policy splits into video prediction × IDM; because the IDM conditions on $\mathbf{o}_{0:l+H}$, actions are inferred from imagined futures, not the present.
• Eq. 2 rectified-flow interpolation: $\mathbf{z}_{t_k}^{k}=t_k\mathbf{z}_1^k+(1-t_k)\mathbf{z}_0^k$ (and the same for $\mathbf{a}$) — linear path from Gaussian noise ($t{=}0$) to clean data ($t{=}1$), shared $t_k$ across both modalities in the base model.
• Eq. 3 loss: $\mathcal{L}(\theta)=\mathbb{E}\big[\tfrac{1}{K}\sum_k w(t_k)\lVert\mathbf{u}_\theta([\mathbf{z}_{t_k}^k,\mathbf{a}_{t_k}^k];\mathcal{C}_k,\mathbf{c},\mathbf{q}_k,t_k)-\mathbf{v}^k\rVert^2\big]$ — chunk-averaged weighted velocity regression under clean-context teacher forcing.
• Eqs 4–6 (Flash): coupled $t_k^{\text{video}}=t_k^{\text{action}}=t_k\sim\mathcal{U}(0,1)$ becomes decoupled $t_k^{\text{video}}=1-\eta,\ \eta\sim\text{Beta}(\alpha,\beta)$, action stays uniform; the interpolation (Eq. 6) is then applied per modality.

Loss decomposition: the single L2 velocity term jointly enforces (a) video-frame denoising and (b) action denoising, coupled through the shared DiT and (in base mode) shared timestep — there is no separate reconstruction or reward term.

6 minimum checks

1. Objective is a proper flow-matching loss — Eq. 3 regresses to $\mathbf{v}^k=$ clean − noise, the standard rectified-flow velocity target; ✓ well-posed.
2. Factorization is exact, not approximate — Eq. 1 is a chain-rule decomposition (video term × IDM term conditioning on the produced future), not an inequality; ✓.
3. Teacher forcing consistency — training conditions on clean $\mathcal{C}_k$ while inference substitutes GT observations into the same context slots (Fig. 14), so train/inference conditioning matches for the video-context path; ✓ (the timestep path is where Flash later fixes a residual mismatch).
4. Flash expectation arithmetic — $\text{Beta}(7,1)$ gives $\mathbb{E}[\eta]=7/8=0.875$, hence $\mathbb{E}[t_k^{\text{video}}]=0.125$ vs $0.5$ coupled; ✓ (biased toward high noise as claimed).
5. Dimensional/horizon consistency — video chunk $K=2$ latent frames matches action horizon; AgiBot $H=48$@30Hz and DROID $H=24$@15Hz both span 1.6s/chunk; ✓ internally consistent.
6. Speedup composition — Table 1 rows are cumulative and multiply plausibly to 38× on GB200 (system ~14.8×, +quant 16.6×, +Flash 38×); ✓ arithmetically coherent, though the final Flash jump is empirical not derived.

---

5. 实验与数据 #



Paper's Figure 8, verbatim (caption: "Seen Task Evaluation. DreamZero effectively learns from diverse data and generalizes to new environments, outperforming VLAs across all task categories. VLAs trained from scratch achieve near-zero success, while pretrained VLAs show modest performance...").

The load-bearing headline: 62.2% vs 27.4%. Notice from-scratch VLAs sit at ≈0% — the gap is not just "DreamZero is better" but "VLAs cannot learn generalizable policies from non-repetitive data at all," which is the paper's core thesis about the how-prior.



Paper's Figure 9, verbatim (caption: "Zero-shot Generalization to Unseen Tasks. DreamZero achieves non-trivial task progress on 10 tasks absent from training, while VLAs struggle across both embodiments.").

39.5% vs 16.3% on tasks whose motions (not just objects) were never seen — e.g. "Remove Hat from Mannequin" 85.7%. This is the strongest evidence that visual planning transfers to novel skills where motion-primitive VLAs collapse to reaching-and-grasping.

Table 3 — DreamZero-Flash (table bussing, varying denoising steps)

Method	Denoising steps	Task Progress	Inference speed	×Speed up
DreamZero	4	83% ± 6.1%	350ms	1×
DreamZero	1	52% ± 10.2%	150ms	2.33×
DreamZero-Flash	1	74% ± 10.1%	150ms	2.33×



The single most important ablation for the core barrier: naive 1-step inference craters (83→52%), but Flash's decoupled schedule recovers most of it (74%) at the same 2.33× speed. This is the empirical proof that the train/test timestep mismatch — not raw step count — was the bottleneck.

Table 4 — Model and Data Ablations (PnP Easy)

Group	Arch	Size	Data	Task Progress
Data diversity	AR	14B	Repetitive	33% ± 4.2%
Data diversity	AR	14B	Diverse	50% ± 6.3%
Model scale	AR	5B	Diverse	21% ± 4.2%
Model scale	AR	14B	Diverse	50% ± 6.3%
Architecture	BD	14B	Diverse	50% ± 14.4%
Architecture	AR	14B	Diverse	50% ± 6.3%



Three causal isolations: diverse data beats repetitive (33→50%) even on simple pick-and-place; 14B beats 5B (50 vs 21%, smaller model hallucinates video); AR ties BD on task progress (both 50%) but is chosen for smoothness + 3–4× KV-cache speed. Note the paper's own prose says larger VLAs get 0% on diverse data, which is in tension with the "VLA 50%±0.0%" rows in the full table — flagged, not resolved.



Paper's Figure 16, verbatim (caption: "Illustration of generated and executed pair... These two examples show scenarios where the video prediction failed and the robot followed the failed plan.").

This substantiates the key diagnostic claim: most failures are video-plan errors, not action-extraction errors. The robot faithfully executes even a wrong imagined plan — evidence that improving the video backbone would directly lift policy performance.

Training recipe & scale #

Stage	Purpose	Data	Steps	Batch	Special technique
Pretrain (AgiBot)	learn joint video+action	~500h, 22 envs, 7.2K eps	100K	128	full DiT + encoders updated; text/image/VAE frozen
Pretrain (DROID)	Franka validation	DROID	100K	128	relative joint actions, idle filtered
Post-train	task-specific FT	12–40h/task	50K/task	—	image-overlay eval variance reduction
Flash	few-step inference	—	final stage	—	$\text{Beta}(7,1)$ decoupled video noise
Cross-embodiment	video-only transfer	72 traj, 10–20 min	10K	—	1:1 mix, video-only objective on new embodiment

Backbone: Wan2.1-I2V-14B-480P. Hyperparameters: $K=2$ latent frames/chunk, $M=4$ chunks, 8-latent-frame context (≈6.6s); 16 diffusion steps for smooth actions (→4 via caching →1 via Flash). GPU hours / MFU: [论文未披露]. LoRA was tried and found suboptimal (full-parameter FT required).

Convergence & stability #

• The shared-timestep base model is used "for faster convergence at the beginning of training"; Flash is applied as a final stage, implying instability if decoupling is introduced early.
• Reducing steps naively degrades action quality because residual video noise propagates into actions — the documented failure mode motivating Flash.
• Action Chunk Smoothing (upsample 2×, Savitzky-Golay window 21 / order 3, downsample) suppresses high-frequency action noise — a stability trick against jittery outputs.

Dataset analysis #

Composition: ~500h AgiBot teleoperation across 22 real-world environment types (homes, restaurants, supermarkets, offices, warehouses, hotels), 7.2K episodes, ~4.4 min/episode, ~42 subtasks/episode. Collection philosophy is breadth-first: 3 coarse tasks/episode, a task retired after 50 episodes to force a long-tail distribution. Contamination: eval sites are in a different geographic location than training — every benchmark is inherently OOD. Synthetic/human data: 12 min egocentric human video + 20 min YAM robot video used video-only for transfer.

---

6. 论证链 #

#	Step	Support (paper-internal)
1	VLAs pretrained on static image-text lack spatiotemporal "how" priors, so they fail on unseen motions.	§1–§2.1 argument; Fig. 9 VLAs ≈16.3% on unseen tasks.
2	A video-diffusion backbone already encodes physical dynamics; jointly predicting video+action reduces policy learning to inverse dynamics on an imagined future (Eq. 1).	§2.2 "Why WAMs"; §3.1 factorization.
3	Autoregressive chunks + GT-KV injection preserve native FPS and eliminate compounding error that plagues bidirectional/pure-AR-video models.	§3.1, Fig. 4/14; App. B Fig. 13.
4	Trained this way on diverse (not repetitive) data, the model generalizes 2× better than VLAs to new envs and unseen tasks.	Fig. 8 (62.2 vs 27.4%), Fig. 9 (39.5 vs 16.3%), Table 4 (diverse 50 > repetitive 33%).
5	Because actions are read from the visual plan, video-only data (even human egocentric) transfers skills without action labels; new embodiments adapt in 30 min.	Q4/Q5, Table 2 (38.3→55.4%), Fig. 12.
6	The one remaining blocker (iterative denoising latency) is removed by a 38× optimization stack, whose model-level piece (Flash decoupled noise) fixes the few-step train/test mismatch.	§3.2, Table 1 (38×), Table 3 (Flash 1-step 74 vs 52%).

---

7. 实现 cross-reference #

Open-sourced weights, inference code, and benchmark runners at https://github.com/dreamzero0/dreamzero (per §1 contributions). Training code openness beyond inference is not fully specified in the L1; the recipe is otherwise reconstructable from the two verbatim algorithms.

• 核心技术壁垒 — DreamZero-Flash decoupled noise schedule: implemented in the training loop's timestep sampling. Per Algorithm 1 (App. C), the Flash branch samples t_vid ~ Beta(7,1) and t_act ~ U(0,1) before the Eq. 2 interpolation, versus the coupled t_vid ← t_k, t_act ← t_k default. Getting $\alpha>\beta$ right (mass near $\eta\approx1$ ⇒ high video noise) and applying Flash only as a final training stage is the easy-to-miss piece that separates 74% from 52% at 1 step.
• 关键实现细节 (1) GT-KV injection: Algorithm 2 (App. C) inference loop — after async-executing the filtered action, real observations are re-encoded (z_real ← VAE(o_real)) and written back with update=True while the predicted video latent is discarded. Skipping this reintroduces compounding error.
• 关键实现细节 (2) DiT caching threshold: reuse the cached velocity when CosSim(v_prev, v_last) > ε, cutting effective DiT steps 16→4; combined with CFG parallelism across 2 GPUs (−47% per-step) and NVFP4 quantization (QKV/Softmax kept in FP8, LayerNorm/RoPE in FP16) on Blackwell. These are the non-obvious levers behind Table 1's cumulative speedups.

Formal implementation citations: Algorithm 1 (training) and Algorithm 2 (inference) are reproduced verbatim in L1 App. C; Eqs 1–6 in §3.1 / App. D.3. No source file:line anchors are available in the L1 — [实现未公开] for line-level pointers pending repo inspection.