§1 TL;DR #
Seedance 1.5 pro 基于 dual-branch MMDiT 实现 native 音视频联合生成，通过 multi-stage data pipeline + SFT + RLHF 后训练 + >10× distillation 加速，在中文语境的唇语同步、方言支持和镜头调度上建立竞争优势。
§2 痛点 / 方法 / 结果 #
Q1 痛点 #
视频生成已从视觉质量竞赛进入"实用生产工具"阶段，但音视频一致性仍是核心瓶颈：
- 唇形-语音时序错位（ventriloquism effect）
- 多语言/方言场景下的 prosody 失真
- 缺乏端到端 joint generation — 现有方案多为 video-first + audio post-processing
Q2 方法 #
四大技术贡献：
| 贡献 | 内容 |
|---|
| Data Framework | Multi-stage curation pipeline + captioning system (视频/音频双模态) + curriculum scheduling |
| Architecture | Dual-branch MMDiT: video branch + audio branch + cross-modal joint module |
| Post-training | SFT on high-quality AV data → RLHF with multi-dimensional reward model (motion + aesthetics + audio fidelity)；RLHF pipeline 3× training speed |
| Acceleration | Multi-stage distillation (NFE reduction) + quantization + parallelism → >10× end-to-end |
核心技术壁垒: Cross-modal joint module 在 MMDiT 架构内实现 deep interaction（非简单拼接或后处理），使音视频在 denoising 过程中逐步对齐。具体机制论文未披露。
Q3 结果 #
- T2V instruction following: leading position vs Kling 2.5/2.6, Veo 3.1, Seedance 1.0
- Audio-visual sync: surpasses Veo 3.1 and Kling 2.6 in lip-sync accuracy
- Chinese audio: consistent advantage over Veo 3.1 in dialect/dialogue
- Audio expressiveness: Sora 2 more vivid emotionally; Seedance 1.5 pro more balanced/controlled
- Inference: >10× acceleration over base model
- Deployment: Volcano Engine (Doubao/Jimeng platforms)
§3 架构 / 方法图 #
flowchart TB
subgraph Training["Training Pipeline"]
D["Multi-stage Data Curation"] --> PT["Video-Audio Joint Pre-training"]
PT --> SFT["SFT on High-quality AV"]
SFT --> RLHF["RLHF (multi-dim reward: motion + aesthetics + audio)"]
end
subgraph Architecture["Dual-Branch MMDiT"]
TE["Text Encoder"] --> VB["Video Branch (DiT)"]
TE --> AB["Audio Branch (DiT)"]
VB <-->|"Cross-Modal Joint Module"| AB
VB --> VD["Video Decoder"]
AB --> AD["Audio Decoder"]
end
subgraph Inference["Inference Pipeline"]
UP["User Prompt"] --> PE["Prompt Engineering"]
PE --> TE2["Text Encoder"]
TE2 --> DIT["Video-Audio Joint DiT"]
DIT --> REF["Refiner"]
REF --> OUT["Video + Audio Output"]
end
subgraph Accel["Acceleration (>10×)"]
DIST["Multi-stage Distillation (NFE↓)"]
QUANT["Quantization"]
PAR["Parallelism"]
end
§4 作者证明 #
无形式化作者证明 — 仅实证
论文为产品技术报告，无任何数学公式、算法伪代码或理论分析。所有 claims 基于人工评估（SeedVideoBench 1.5, proprietary benchmark）。
期望但缺失的形式化内容 #
| 期望 | 现状 |
|---|
| Audio representation (waveform? spectrogram? codec tokens?) | 未披露 |
| Cross-modal attention 具体计算方式 | 仅说 "deep cross-modal interaction" |
| Distillation loss function 和 NFE 曲线 | 仅说 ">10×" |
| Model size / layer count / hidden dim | 外部信息约 4.5B, 论文未写 |
| RLHF reward model architecture | 仅说 "multi-dimensional" |
| Inference latency breakdown | 未披露 |
评估可信度分析 #
- Benchmark proprietary: SeedVideoBench 1.5 未公开 — 无法第三方复现
- Human evaluators: "professional film directors" + "experts from film production" — 专业但主观
- Comparative claims 有选择性: 承认 Sora 2 expressiveness 更强 → 部分诚实
- No quantitative tables: 所有比较结果以 Figure (bar chart) 形式呈现，无 exact numbers
- Self-critique of competitors: 批评其他模型 "slow-motion trick" — 未提供证据
- 197 authors, no attribution: 最大规模作者列表但零贡献声明
§5 实验与数据 #
训练配置 #
| Stage | Details |
|---|
| Pre-training | Large-scale mixed-modality (video + audio + text); curriculum scheduling |
| SFT | High-quality audio-video pairs (dataset size undisclosed) |
| RLHF | Multi-dimensional reward model; 3× pipeline speed improvement via infra optimization |
| Distillation | Multi-stage, NFE reduction (references: Mean Flows, Hyper-SD, RayFlow) |
评估框架 (SeedVideoBench 1.5) #
Video 维度: Motion Quality (stability + vividness), Prompt Following (intent-aligned), Visual Aesthetics
Audio 维度: Audio Prompt Following, Audio Quality, Audio-Visual Sync, Audio Expressiveness
关键比较结果 #
Video (T2V, absolute 5-point Likert, Fig. 3):
- Instruction following: Seedance 1.5 > Veo 3.1 > Kling 2.6 > Seedance 1.0
- Motion dynamics: competitive with Veo 3.1
Audio (GSB pairwise, Fig. 5-6):
- vs Veo 3.1: Win on Chinese vocal, lip-sync; Lose on expressiveness
- vs Kling 2.6: Win on AV synchronization
- vs Sora 2: Lose on emotional expressiveness; Win on stability/control
- vs Wan 2.5: Win overall
Deployment #
Platform: Volcano Engine (火山引擎), integrated into Doubao (豆包) and Jimeng (即梦) by Dec 2025.
§6 论证链 #
| Step | 论据 | 证据 | 结论 |
|---|
| 1 | Native joint generation > post-processing | Architecture: dual-branch MMDiT with cross-modal module (vs separate video + TTS) | 音视频在 denoising 中共同优化 → 更好时序对齐 |
| 2 | Post-training 提升质量 | SFT + RLHF → human eval shows improvement over Seedance 1.0 (Fig. 3-4) | 证据: 相比前代全面提升 |
| 3 | 中文语境优势 | Dialect support (Sichuanese, Cantonese, etc.) + lip-sync accuracy | 超越 Veo 3.1 在中文生成 |
| 4 | 加速不损质量 | >10× acceleration claimed "while preserving model performance" | 实用部署可行 (具体数据未给) |
§7 实现 cross-reference #
代码: 未开源。模型通过 Volcano Engine API 商业提供。
关键实现细节:
- Multi-task pre-training: 同一模型同时支持 T2VA, I2VA, T2V, I2V — 通过 task token 或 conditioning 切换。
- RLHF 加速: "nearly 3× improvement in training speed" 通过基础设施优化（具体方法未披露，可能涉及 selective rollout、gradient accumulation 优化或 reward model 推理加速）。
参考实现 (related):
- MMDiT backbone: Esser et al. 2024 (SD3)
- Distillation: Mean Flows (Geng et al. 2025), Hyper-SD (Ren et al. 2025), RayFlow (Shao et al. 2025)
- RLHF: Flow-GRPO (Liu et al. 2025), DanceGRPO (Xue et al. 2025), RewardDance (Wu et al. 2025)
[实现未公开]