SageAttention3 — L2 Deep Analysis #

§1 TL;DR #

Microscaling FP4 attention with two-level quantization achieves 1038 TOPS (5× FlashAttention2) on RTX 5090; trainable INT8 attention keeps $dO \cdot V^\top$ in FP16, delivers lossless fine-tuning at 1.67× speedup but degrades pretraining convergence.

§2 Q1 · Q2 · Q3 #

Q1 — 痛点 #

Attention is the quadratic-complexity bottleneck in Transformer inference and training. Blackwell GPUs provide FP4 Tensor Cores at 1600 TOPS raw throughput (8× over FP16), but exploiting them for attention faces three challenges:

1. C1 — FP4 has only 15 representable values (E2M1). Per-tensor or per-token quantization of Q, K, V destroys accuracy because a single outlier contaminates the scale for the entire group.
2. C2 — Softmax output $\widetilde{P} \in [0,1]$ collapses scale factors. NVFP4 requires scale factors in E4M3 FP8; dividing by max = 6 puts all scales in $[0, 0.167]$, wasting > 95% of E4M3's representable range.
3. C3 — Training gradient sensitivity. Among the five backward matmuls, quantizing $dO \cdot V^\top$ creates error that cascades through $dP \to dS \to dQ, dK$, accumulating over the sequence-length loop.

No prior work applied low-bit attention to training; FlashAttention3's FP8 mode is inference-only and Hopper-only.

Q2 — 方法 #

Problem formulation. Standard FlashAttention tiling produces block-wise attention:

$$S_{ij} = Q_i K_j^\top, \quad P_{ij} = \text{OnlineSoftmax}(S_{ij}), \quad O_{ij} = P_{ij} V_j$$

The objective is to replace both matmuls ($QK^\top$ and $PV$) with FP4 Tensor Core instructions while preserving output cosine similarity > 99%.

Inputs / outputs of one step: a FlashAttention tile $(Q_i, K_j, V_j)$ in FP16 → quantized FP4 operands + FP8 scales → FP32 accumulator output $O_{ij}$.

The one novel mechanism — two-level quantization for $\widetilde{P}$.

Aspect	Before (direct FP4 microscaling)	After (two-level quantization)
Scale-factor range for $\widetilde{P}$	$s_P \in [0, 0.167]$ (E4M3 underutilized)	$s_{P_2} \in [0, 448]$ (full E4M3 range)
CosSim on CogVideoX	93.32%	99.52%
Extra per-token cost	None	One FP32 rowmax + division
Mechanism	$\phi(\widetilde{P})$ directly	Stretch $\widetilde{P} / s_{P_1}$ to $[0, 2688]$, then $\phi$

The constant $448 \times 6 = 2688$ is hardware-derived: 448 = E4M3 max, 6 = E2M1 FP4 max. Stretching to their product ensures the second-level scale factors saturate E4M3's full dynamic range.

核心技术壁垒: The naive FP4 microscaling bottleneck lies not in the 4-bit data representation itself but in the scale factor's format (E4M3) being starved of dynamic range by the softmax output's narrow $[0,1]$ domain. The fix — pre-stretching by $\text{rowmax}/(448 \times 6)$ — is zero-cost because it reuses the online softmax's existing row-max computation. The insight is hardware-format-aware: it exploits the specific interplay between E2M1 data and E4M3 scales in NVFP4's instruction format.

Secondary contribution — SageBwd (trainable 8-bit attention): keep $dO \cdot V^\top$ in FP16 (the single accuracy-critical path), quantize remaining 6/7 matmuls to per-block INT8. INT8 outperforms FP8 here because symmetric quantization better matches the bell-shaped distributions of attention matrices.



Paper's Figure 1, verbatim (caption: "The upper left figure shows the kernel speedup on RTX5090. The other two figures show the end-to-end inference speedup of generating a video using HunyuanVideo on RTX5090.").

The bar chart (upper left) places SageAttention3 at ~1038 TOPS versus FlashAttention2 at ~200 TOPS, visualizing the 5× gap. The end-to-end timelines (lower panels) show that this kernel-level speedup translates into concrete wall-clock savings for HunyuanVideo generation. Note that FlashAttention3 is absent because it cannot run on Blackwell consumer GPUs (Hopper-only).

Q3 — 结果 #

Metric	Value
FP4 inference kernel TOPS (RTX 5090, hd128)	1038 TOPS (5× FA2, 11× xformers)
FP4 attention CosSim (CogVideoX)	99.52%
End-to-end video gen quality (HunyuanVideo CLIPSIM)	0.1866 vs 0.1838 FP16
SageBwd training speedup (RTX 4090)	1.67× FA2, 3× xformers
Fine-tuning accuracy (Qwen2.5-3B, GSM8K)	0.607 vs 0.601 BF16
Pretraining convergence	Slower than BF16 — not lossless

§3 架构 / 方法図 #



Paper's Figure 2, verbatim (caption: "Workflow of microscaling FP4 attention.").

The workflow proceeds left-to-right: Q and K are microscaling-quantized ($\phi$) to FP4 with E4M3 scale factors at 1×16 block granularity, then multiplied via FP4MM to produce score matrix S in FP32. Online softmax yields $\widetilde{P}$, which undergoes two-level quantization (per-token FP32 stretch → microscaling FP4) before a second FP4MM with quantized V produces the output tile. The FP32 accumulator ensures numerical stability across the tiling loop.

Quantization pipeline (both matmuls):

Per-block FP4 microscaling ($\phi$): $s_{ij} = \max(|X_{ij}|)/6, \quad \hat{X}_{ij} = \lceil X_{ij}/s_{ij} \rfloor$, where block size $n = 16$ (NVFP4), scale in E4M3.

First matmul ($QK^\top$):

$$S_{ij} = \texttt{FP4MM}(\hat{Q}_i, s_Q, \hat{K}_j, s_K) + \texttt{GEMV}(\bar{q}_i, K_j^\top)$$

The GEMV corrects for the smooth-Q mean subtraction. Online softmax produces $\widetilde{P}_{ij}$.

Two-level quantization of $\widetilde{P}$:

$$s_{P_1} = \text{rowmax}(\widetilde{P}_{ij}) / (448 \times 6), \quad s_{P_2}, \hat{P} = \phi(\widetilde{P}_{ij} / s_{P_1})$$

Second matmul ($PV$):

$$O_{ij} = \texttt{FP4MM}(\hat{P}, s_{P_2}, \hat{V}, s_V) \times s_{P_1}$$

The complete algorithm includes three hardware optimizations: (1) column permutation of K to match the FP4 accumulator layout without thread shuffles, (2) fused quantization with online softmax reusing row-max reductions (10% kernel speedup), and (3) producer-warp ping-pong scheduling to overlap MatMul with global memory stores under tight register constraints.



Paper's Figure 3, verbatim (caption: "Analysis of the benefit of two-level quantization.").

Sub-figure (a) shows $\widetilde{P}$'s values concentrated in $[0,1]$. Sub-figures (b) vs (c) are the critical comparison: direct quantization confines $s_P$ to a narrow band near zero (wasting E4M3 range), while two-level quantization spreads scale factors across E4M3's full representable range. The resulting error distributions (d vs e) show an order-of-magnitude reduction in both scale-factor error and final quantization error. This single figure is the strongest visual argument for the paper's core contribution.

§4 作者证明 #

记号表 #

Symbol	Meaning	Domain
$\phi(\cdot)$	FP4 microscaling quantization	$\mathbb{R}^{N \times d} \to (\text{FP4}^{N \times d}, \text{E4M3}^{N \times d/16})$
$\phi^{-1}(\hat{X}, s)$	Dequantization	FP4 + E4M3 → $\mathbb{R}$
$s_{ij}$	Per-block (1×16) scale factor	E4M3 FP8 (max 448)
$\hat{X}_{ij}$	Quantized block	E2M1 FP4 (max 6, 15 values)
$n$	Microscaling block size	16 (NVFP4) or 32 (MXFP4)
$s_{P_1}$	Per-token first-level scale	FP32
$s_{P_2}$	Per-block second-level scale	E4M3 FP8
$\psi(\cdot)$	INT8 per-block quantization	$\mathbb{R}^{B \times d} \to (\text{INT8}^{B \times d}, \text{FP32})$
$\bar{q}_i$	Per-block Q mean (smooth-Q)	FP16, shape $1 \times d$
$K_m$	Global K mean (smooth-K)	FP16, shape $1 \times d$

方程物理意义 #

1. Eq. 1–2 ($\phi / \phi^{-1}$): Block-wise FP4 quantization with scale = max/6. The divisor 6 is the hardware max of E2M1; the 1×16 granularity bounds outlier contamination to 16 elements per scale factor.
1. Eq. 3 (FP4MM): Single hardware instruction fusing dequantization + matmul. Achieves 1600 TOPS raw because the Blackwell Tensor Core processes 4-bit operands at 2× the throughput of INT8.
1. Eq. 4 (attention pipeline): Applies $\phi$ to both matmuls in attention. The smooth-Q correction ($\bar{q}_i$ GEMV) compensates for the mean-subtraction required by microscaling, adding $O(B_q \times d)$ work — negligible versus the $O(B_q \times B_{kv} \times d)$ FP4MM.
1. Eq. 5 (two-level quantization): $\widetilde{P} \approx \hat{P}_2 \times s_{P_2} \times s_{P_1}$. Two multiplicative scales: $s_{P_1}$ (FP32, per-token) stretches $[0,1]$ to $[0,2688]$; $s_{P_2}$ (E4M3, per-16-block) provides fine-grained adaptation. The product $448 \times 6$ is the maximum value representable by the product of one E4M3 and one E2M1 number — this is the only choice that fully saturates both formats.
1. Eq. 6–7 (INT8 forward): Standard per-block INT8 quantization $\psi$ for training forward pass. $\widetilde{P}$ uses per-token quantization that reuses online softmax's row-max, avoiding a separate reduction pass.
1. Eq. 8 (backward matmuls): Five matmuls identified; keeping $dO \cdot V^\top$ in FP16 breaks the error accumulation chain at the earliest dependency point ($dP \to dS$). The remaining four matmuls use per-block INT8 and accumulate errors independently in $dV$, $dQ$, $dK$.

6 Checks #

1. Notation consistency: $\phi$ used consistently for FP4 microscaling (Eqs. 1–5); $\psi$ for INT8 (Eqs. 6–7). Block subscript convention ($i, j$ for tiles) is coherent with FlashAttention literature. Bold/non-bold distinction (matrix vs element) maintained throughout algorithms.
1. Dimension analysis: $\phi$ maps $\mathbb{R}^{1 \times 16} \to (\text{FP4}^{1 \times 16}, \text{E4M3}^{1 \times 1})$ — scale vector has $d/16$ entries per row, matching FP4MM's expected scale operand shape. Output $O \in \mathbb{R}^{N \times d}$ matches input dimensionality. Two-level quantization: $s_{P_1} \in \mathbb{R}^{B_q \times 1}$ (per-row), $s_{P_2} \in \text{E4M3}^{B_q \times B_{kv}/16}$ (per-block) — dimensions compose correctly in the final dequantization.
1. Boundary case — $\widetilde{P} = 0$: When attention is uniform, $\text{rowmax}(\widetilde{P}) \approx 1/N$ and $s_{P_1} \approx 1/(2688N)$. For practical $N \leq 10^6$, this is $\sim 10^{-10}$ — well within FP32 range. When one token dominates (sparse attention), $s_{P_1} \approx 1/2688$, also safe. Edge case: if an entire row of $\widetilde{P}$ is zero (masked), $s_{P_1} = 0$ and the product is correctly zero.
1. Monotonicity: Finer block granularity (16 vs 32) → lower quantization error (Table 1a: NVFP4 99.52% > MXFP4 98.37%). Two-level quantization → wider scale utilization → lower error (Table 1b: 99.52% vs 93.32%). Both effects are monotonic and align with quantization theory.
1. Consistency with prior results: SageAttention2's smooth-Q/K techniques remain effective at FP4 — the paper inherits them without modification. INT8 outperforming FP8 for symmetric distributions aligns with quantization theory (symmetric formats match bell-shaped data without wasting a sign bit on all-positive or all-negative tails).
1. Reproducibility from equations: Eqs. 1–5 + Algorithm 1 fully specify the FP4 forward pass. The three hardware optimizations (§3.3) require CUTLASS/PTX knowledge but are described precisely (permutation pattern for K, fusion of quantization with softmax max-reduction, producer-warp ping-pong). Backward pass (Algorithm 3) is fully specified including the single FP16 matmul choice.

Formal guarantees #

The paper provides a quantization error bound for two-level quantization in Appendix A.5, proving that the effective dynamic range of E4M3 scale factors is expanded from $[0, 0.167]$ to $[0, 448]$, reducing the worst-case representation error. This is an error-bound analysis, not a convergence theorem.

无形式化收敛证明 — 仅实证 for the training contribution (SageBwd). A desirable guarantee would be: given pretrained weights $\theta_0$ and fine-tuning data $\mathcal{D}$, bound the distance $\|\theta^{\text{INT8}}_T - \theta^{\text{FP16}}_T\|$ as a function of bit-width, sequence length $N$, and head dimension $d$. The paper provides only empirical evidence (Table 3, Fig. 8) without such a bound.

§5 实验与数据 #

Inference kernel speed #



Paper's Figure 4, verbatim (caption: "Speed comparison between SageAttention3 and Baselines (RTX5090, headim=128).").

SageAttention3 peaks at 1038 TOPS on RTX 5090 (headim = 128), representing ~65% utilization of the raw 1600 TOPS FP4 Tensor Core throughput. The ~35% overhead comes from in-kernel quantization ($\phi$ for Q, K, V, P), online softmax, two-level scaling, and memory traffic for scale factors. FlashAttention2 caps at ~200 TOPS (FP16 Tensor Cores). Additional speed figures for headim = 64 (Fig. 5) and SageBwd on RTX 4090 (Figs. 6–7) show consistent speedup patterns.

Inference quality (end-to-end) #

Model	Task	SA3 (4-bit)	FP16 baseline	Delta
CogVideoX	CLIPSIM ↑	0.1881	0.1865	+0.0016
HunyuanVideo	VQA-t ↑	75.440	78.891	−3.45
HunyuanVideo	FScore ↑	1.232	1.479	−0.247
Mochi	CLIPSIM ↑	0.1800	0.1828	−0.0028
Flux	FID ↓	162.121	162.812	−0.69 (better)
SD3.5	CLIP ↑	32.01	31.93	+0.08

Quality is largely maintained across models, but HunyuanVideo shows non-trivial VQA-t (−3.45) and FScore (−0.247) degradation. The "plug-and-play" claim holds on aggregate but has model-dependent caveats.

Training results #



Paper's Figure 8, verbatim (caption: "Pretraining and Finetuning loss curves of BF16 and 8-bit attention.").

Sub-figures (a)–(b) show pretraining: 8-bit attention consistently converges slower, with a visible gap that does not close within the training budget. Sub-figures (c)–(e) show fine-tuning: 8-bit curves overlay BF16 almost exactly. This asymmetry suggests pretrained weights provide an error-absorbing landscape that makes attention quantization noise inconsequential during fine-tuning, while training from scratch lacks such regularization.

Model	Method	GSM8K ↑	DROP ↑	MMLU ↑	HELLASWAG ↑
Qwen2.5-1.5B	BF16	0.521	0.733	0.569	0.905
Qwen2.5-1.5B	SageBwd	0.520	0.734	0.574	0.911
Qwen2.5-3B	BF16	0.601	0.785	0.640	0.944
Qwen2.5-3B	SageBwd	0.607	0.782	0.653	0.943
Llama3.2-1B	BF16	0.259	0.641	0.464	0.828
Llama3.2-1B	SageBwd	0.268	0.637	0.458	0.823

All fine-tuning metrics within ±0.013 of BF16 — consistent with the "lossless" claim for fine-tuning.

Training recipe & scale #

This paper is not primarily a training recipe paper; training experiments validate the feasibility of low-bit attention during training.

Stage	Purpose	Model	Hardware	Technique
Pretraining	Validate convergence	Qwen2.5, Llama3.2	RTX 4090	6/7 INT8 matmuls, $dO V^\top$ FP16
Fine-tuning	Validate quality	Same	Same	Same

• Total tokens per stage: [论文未披露]
• GPU hours and MFU: [论文未披露]
• Training data composition: [论文未披露] — standard instruction-following for fine-tuning, standard corpora for pretraining
• Critical hyperparameter: the binary choice of which backward matmul stays FP16
• Stability: smooth-K (mean subtraction from SageAttention), per-token quantization of $\widetilde{P}$ reusing softmax max

Convergence & stability #

• Learning curve shape: Fine-tuning — indistinguishable from BF16 (Fig. 8 c–e). Pretraining — consistently higher loss with a fixed additive gap rather than divergence (Fig. 8 a–b).
• Sensitivity: The critical binary choice is $dO \cdot V^\top$ in FP16 vs INT8. Moving this single matmul to INT8 drops $dQ$ CosSim from 99.77% to 97.47% (Table 1c). No intermediate option exists.
• Where does full-precision help? Pretraining from scratch — accumulated quantization error compounds into weight updates more destructively without a pretrained initialization to anchor the trajectory.

NVFP4 vs MXFP4 ablation #

Format	Block size	Scale format	CosSim	RMSE
MXFP4	1×32	E8M0	98.37%	0.994
NVFP4	1×16	E4M3	99.52%	0.201

NVFP4 wins by 1.15pp CosSim and 5× lower RMSE. The finer block size (16 vs 32) is the dominant factor; E4M3's mantissa bits provide a secondary benefit over E8M0's exponent-only format.

INT8 vs FP8 for training #

INT8 SageBwd outperforms FP8 SageBwd. INT8's symmetric $[-127, 127]$ range aligns with the roughly bell-shaped distributions of attention matrices ($Q, K, dS$), while FP8's asymmetric exponent range wastes bits on magnitudes that rarely appear. This contradicts the broader industry trend toward FP8 training (e.g., DeepSeek-V3) but is specific to attention-only quantization where distributions are well-behaved.

Dataset analysis #

Evaluation benchmarks are standard and well-established:

• Video: VBench metrics (CLIPSIM, CLIP-T, VQA-a, VQA-t, FScore) on CogVideoX, HunyuanVideo, Mochi
• Image: FID, sFID, CLIP score, ImageReward on Flux, SD3.5
• LLM fine-tuning: GSM8K, DROP, MMLU, HELLASWAG
• Contamination check: [论文未披露] — standard benchmarks presumed safe
• Annotation: Not applicable (all automated metrics)

§6 论证链 #

Step	Claim	Evidence	Depends on
1	FP4 Tensor Cores deliver 8× raw speedup over FP16 on Blackwell	Hardware spec: 1600 vs 200 TOPS (RTX 5090 whitepaper)	—
2	Naive FP4 quantization of attention destroys accuracy (93.32% CosSim)	Table 1(b) direct-quantization row; Fig. 3(b,d) scale-factor crowding	Step 1 (motivates why FP4 cannot be used naively)
3	Two-level quantization restores accuracy to 99.52% CosSim	Table 1(b) two-level row; Fig. 3(c,e); Appendix A.5 error bound	Step 2 (solves the accuracy gap)
4	NVFP4 (block-16, E4M3) outperforms MXFP4 (block-32, E8M0)	Table 1(a): 99.52% vs 98.37% CosSim, 0.201 vs 0.994 RMSE	Step 3 (format choice within two-level framework)
5	Combined FP4 attention achieves 1038 TOPS with maintained quality	Fig. 4 (kernel speed); Table 2 (end-to-end across 7 models)	Steps 3 + 4 + §3.3 hardware optimizations
6	Among 5 backward matmuls, $dO \cdot V^\top$ is uniquely accuracy-critical	Table 1(c): FP16 raises $dQ$ CosSim from 97.47% to 99.77%	Independent (error propagation analysis)
7	SageBwd (6/7 INT8 + 1 FP16) achieves lossless fine-tuning	Table 3: ±0.013 across 3 models × 4 benchmarks; Fig. 8(c–e)	Step 6 (design choice validated)
8	SageBwd fails for pretraining	Fig. 8(a–b): persistent loss gap vs BF16 that does not close	Step 7 (same method, different regime reveals limitation)

§7 实现 cross-reference #

SageAttention3 (FP4 inference):

• Implementation: CUTLASS + CUDA
• Repository: thu-ml/SageAttention on GitHub (open-source)
• FP4 kernels require Blackwell GPU (SM100+)
• The three hardware optimizations (K permutation, fused quantization, producer-warp ping-pong) use CUTLASS warp-specialized kernel templates and PTX-level register management

SageBwd (INT8 training):

• Implementation: OpenAI Triton
• Forward: Algorithm 2 — fully specified, Triton-implementable with standard tl.dot for INT8
• Backward: Algorithm 3 — fully specified; the FP16 $dO \cdot V^\top$ path uses Triton's native FP16 matmul

关键実装細節 #

1. Smooth-Q correction requires a separate GEMV: Subtracting $\bar{q}_i$ from $Q_i$ before quantization means the FP4MM computes $(Q_i - \bar{q}_i)K_j^\top$, requiring a post-hoc correction $+ \bar{q}_i K_j^\top$ via GEMV on FP16. This is $O(B_q \times d)$ per tile — negligible versus the $O(B_q \times B_{kv} \times d)$ FP4MM — but omitting it silently degrades accuracy.
1. K column permutation is fused with K's quantization kernel: The permutation needed to match FP4 accumulator layout to P tile's column ordering is not a separate global-memory pass. It is integrated into the $\phi(K^\top)$ quantization kernel, avoiding an extra memory round-trip that would cost ~5% throughput.

Reproducibility & ecosystem #

• Code: thu-ml/SageAttention (GitHub, open-source). SageAttention and SageAttention2 are already integrated into HuggingFace diffusers and ComfyUI.
• Hardware requirement: SageAttention3's FP4 path requires Blackwell GPUs (RTX 5090 / B200); SageBwd's INT8 path runs on any GPU with INT8 Tensor Cores (Turing+).
• Closest open references: For FP4 matmul, NVIDIA's CUTLASS Blackwell examples; for INT8 training attention, no prior open reference exists — SageBwd is the first.
• Community status: SageAttention2 is widely adopted in inference pipelines. SageAttention3 adoption is pending broader Blackwell availability.