The Llama 3 Herd of Models — L2 #

1. TL;DR #

A dense 405B Transformer trained on 15.6T tokens at $3.8\times10^{25}$ FLOPs, deliberately choosing a boring architecture + SFT/rejection-sampling/DPO alignment over MoE + RL. The bet: quality comes from data curation, scale, and complexity management, not architectural novelty.

2. Q1/Q2/Q3 #

Q1 — 痛点. Frontier-quality foundation models (GPT-4 class) were closed, and the open recipes that existed (Llama 2, 1.8T tokens, complex RLHF) trailed badly. The open question: can a simple, reproducible recipe reach GPT-4 parity if you scale data and compute correctly and forecast the right model size before committing a $3.8\times10^{25}$-FLOP run? Two sub-pains: (a) scaling laws predict next-token loss, not downstream accuracy, so budget→size→benchmark mapping is unreliable; (b) at 16K-GPU scale a single fault stalls the whole synchronous job.

Q2 — 方法. Three "levers" made explicit — data (15T-token curated multilingual corpus with a fixed 50/25/17/8 knowledge/reasoning/code/multilingual mix), scale (compute-optimal 405B chosen via a two-step scaling law), and managing complexity (dense Transformer + GQA; alignment via SFT + rejection sampling + DPO instead of PPO). The pre-training recipe is three staged phases (initial → long-context → annealing); post-training is six iterative rounds of RM→SFT→DPO with per-capability synthetic-data pipelines.

核心技术壁垒: the two-step downstream-scaling-law forecast — first fit compute→optimal-token-count as $N^\star(C)=AC^\alpha$ with $(\alpha,A)=(0.53,0.29)$, then fit a sigmoidal map from normalized NLL to task accuracy calibrated against the older Llama 2 family. This lets Meta commit to "405B on 16.55T tokens" and predict ARC-Challenge accuracy four orders of magnitude out before spending the flagship budget. Reproducing it requires the small-model IsoFLOPs sweep plus a family of higher-compute anchor models — the piece nobody else has.

Q3 — 结果. Llama 3 405B reaches GPT-4-class scores (MMLU 87.3, HumanEval 89.0, GSM8K 96.8, MATH 73.8; Table 2) and the 8B/70B are best-in-class for their size. The two-step scaling-law only slightly underestimated the final flagship accuracy. Achieved >90% effective training time on 16K H100s despite 419 unexpected interruptions in 54 days.

3. 架构 / 方法图 #

Paper's Figure 1, verbatim (caption: "Illustration of the overall architecture and training of Llama 3. Llama 3 is a Transformer language model trained to predict the next token of a textual sequence.").

The deliberate anticlimax of this figure IS the argument: no MoE, no exotic attention — just token embeddings → stacked (self-attention + FFN) → next-token, with autoregressive decoding. The only Llama-2 deltas are GQA (8 KV heads), a cross-document attention mask, a 128K-token tokenizer (3.17→3.94 chars/token compression), and RoPE base $\theta=500{,}000$. "Managing complexity" is enforced at the architecture level so training stays stable at 405B.

Paper's Figure 7, verbatim (caption: "Illustration of the overall post-training approach for Llama 3. Our post-training strategy involves rejection sampling, supervised finetuning, and direct preference optimization.").

The alignment engine: a reward model scores $K$ (10–30) generations per prompt (rejection sampling) to produce SFT data; the SFT model is then aligned with DPO; the best DPO model feeds the next round's generations. Note the feedback edge "Best model for next round" — this is the iterative-rounds mechanism (six cycles) that keeps preference data on-policy.

The one algorithm-level novelty vs. the prior method:

before (Llama 2 / standard RLHF)	after (Llama 3)
PPO on-policy RL for alignment	DPO (offline pair loss) — less compute, better IFEval
Reward-margin term in RM loss	margin term removed (diminishing returns after data scaling)
Pairs (chosen, rejected)	triples (edited > chosen > rejected) via an annotator edit step
DPO loss over all tokens	mask formatting/header/termination tokens + add 0.2·NLL on chosen

4. 作者证明 #

无形式化作者证明 — 仅实证（核心方法为工程/配方，非定理）。以下是论文赖以立论的经验性/半形式化关系。

Notation table.

符号	含义
$C$	pre-training compute budget (FLOPs)
$N^\star(C)$	compute-optimal number of training tokens at budget $C$
$A,\alpha$	scaling prefactor / exponent, fitted $(0.29,0.53)$ (Fig.3: $A=0.299,\alpha=0.537$)
$S$	observed benchmark score (accuracy/EM)
$N$	benchmark sample size
$PP,V,M$	pipeline ranks / interleaved stages per rank / total micro-batches

Load-bearing equations & 物理意义.

1. Compute-optimal token law: $$N^{\star}(C) = A C^{\alpha}$$ Sub-linear ($\alpha\approx0.53$) growth means each extra order of compute warrants ~$10^{0.53}\approx3.4\times$ more tokens, not $10\times$ — this is why the flagship is 405B/16.55T rather than a much larger, undertrained model.
1. Benchmark confidence half-width (binomial): $$CI(S) = 1.96 \times \sqrt{\frac{S(1-S)}{N}}$$ Standard-normal 95% interval; the authors flag it is a lower bound on true variation since subsampling is not the only noise source.
1. Pipeline bubble ratio: $$\text{bubble ratio} = \frac{PP-1}{V \cdot M}$$ Idle-time fraction shrinks with interleaving $V$ and micro-batch count $M$; the "make $N$ tunable" schedule fix (Fig.6) exists to relax the $N=PP$ / $N=M$ constraints so $M$ can grow.

6 minimum checks.

1. Assumption of the token law: a clean power-law holds between $6\times10^{18}$ and $10^{22}$ FLOPs — extrapolated 4 orders of magnitude to $3.8\times10^{25}$. Breaks if the loss-vs-token curvature changes off-regime.
2. Sigmoidal NLL→accuracy step leans on Llama 2 anchors; if Llama 2 sat in a different loss regime the calibration would bias the forecast.
3. IsoFLOPs flattening (Fig.2) is the stated justification that 405B is "safe" — the claim is that near the parabola minimum, size/token trade-off barely moves loss.
4. Robustness-as-implicit-guarantee: §5.1.2 shows MMLU scores stable across label/order/format perturbations, standing in for a formal generalization argument.
5. DPO stabilizers are ablation-backed, not proven: token masking + 0.2·NLL are justified by observed "tail repetition"/logprob-collapse, not a convergence theorem.
6. Annealing value is scale-dependent: +24.0%/+6.4% (GSM8k/MATH) at 8B, negligible at 405B — an empirical, not theoretical, result about in-context capability saturating.

A desirable-but-absent guarantee: a bound on the forecast error of the two-step scaling law (they only report "slightly underestimates").

5. 实验与数据 #



Paper's Figure 2, verbatim (caption: "Scaling law IsoFLOPs curves between $6\times10^{18}$ and $10^{22}$ FLOPs. The loss is the negative log-likelihood on a held-out validation set.").

Each parabola is one compute budget; the pink minima are the compute-optimal points fed into Fig.3's power-law fit. The key visual claim: as compute grows (darker curves), the parabola flattens near its minimum — the empirical backing for choosing 405B without fear of mis-sizing.

Scaling-law forecast (Fig.4). The two-step NLL→accuracy pipeline on ARC-Challenge extrapolates over four orders of magnitude and only slightly underestimates the delivered Llama 3 405B score — the paper's single most load-bearing predictive result, since it justified the entire flagship budget.

Headline benchmarks (Table 2, post-trained).

Benchmark	L3 8B	L3 70B	L3 405B	GPT-4 (0125)	GPT-4o	Claude 3.5 Sonnet
MMLU (5-shot)	69.4	83.6	87.3	85.1	89.1	89.9
HumanEval	72.6	80.5	89.0	86.6	90.2	92.0
GSM8K	84.5	95.1	96.8	94.2	96.1	96.4
MATH	51.9	68.0	73.8	64.5	76.6	71.1
GPQA	32.8	46.7	51.1	41.4	53.6	59.4

405B beats GPT-4 (0125) on MMLU/HumanEval/GSM8K/MATH/GPQA and trails GPT-4o/Claude 3.5 on the hardest reasoning (GPQA, MATH). Where it loses: consistently to GPT-4o/Claude on GPQA and to Claude on HumanEval — the frontier-reasoning gap the paper is candid about.

Reliability at scale (Table 5). Over a 54-day snapshot: 466 interruptions (47 planned, 419 unexpected). GPU-attributed causes dominate — faulty GPU 30.1%, HBM3 17.2%, SRAM 4.5% — ~78% of unexpected stops were hardware. Only 3 needed manual intervention; the rest were automated, enabling >90% effective training time. This table is the load-bearing evidence that a synchronous 16K-GPU dense run is operationally feasible.

Parallelism/MFU (Table 4). 4D parallelism [TP,CP,PP,DP] holds 38–43% BF16 MFU; MFU dips 43%→41% going 8K→16K GPUs (smaller per-DP batch to keep 16M tokens/batch constant), and 38% at 128K context (CP=16).

6. 论证链 #

#	论证步骤	依据（paper-internal）
1	Frontier quality is achievable from a simple recipe if you get data/scale/complexity right.	§1 three-levers framing
2	To pick the right size, forecast downstream accuracy, not just loss: fit $N^\star(C)=AC^\alpha$ then NLL→accuracy sigmoid on Llama 2 anchors.	§3.2.1, Fig.2–4
3	Forecast says 405B on ~16.55T tokens is compute-optimal for the $3.8\times10^{25}$ budget; IsoFLOPs flattening makes 405B a safe choice.	§3.2.1 (Fig.2–3)
4	Realize the run: 4D parallelism at 38–43% MFU on 16K H100s, with automated fault recovery for >90% effective training time.	§3.3, Tab.4–5
5	Stage the data: initial → 800B-token long-context (8K→128K) → 40M-token annealing with Polyak averaging.	§3.4
6	Align simply: six rounds of RM→SFT→DPO; DPO (not PPO) is cheaper and better on IFEval; stabilize with token masking + 0.2·NLL.	§4.1
7	Fill capability gaps with verifiable synthetic data (execution feedback for code, correctness-filtered CoT for math).	§4.3
8	Result: 405B ≈ GPT-4 class; 8B/70B best-in-class; forecast validated.	§5, Tab.2

7. 实现 cross-reference #

[实现未公开] — weights are released (Llama 3.1 8B/70B/405B) but the training/data pipeline is not open-sourced. Closest open references for reproducing the algorithm layer:

• DPO + NLL-regularized loss (§4.1.4): mask formatting tokens, add 0.2·NLL on chosen. Reproducible via TRL's DPOTrainer (rpo_alpha implements the Pang-et-al. NLL term). Verl / OpenRLHF also expose DPO.
• Rejection sampling with PagedAttention (§4.2.2): $K$∈[10,30] generations, RM-select best, max-length gating + KV-page sharing → >2× throughput. Reproducible on vLLM (PagedAttention upstream).
• Execution-feedback code synthesis (§4.3.1): generate problem → solve → static/dynamic (linter + containerized unit tests) → iterative self-correct; only all-pass dialogs kept (~20% self-corrected). No public artifact; must be rebuilt.

核心技术壁垒 (§7 dedicated paragraph). The un-replicable asset is not any single loss or kernel — it is the calibrated two-step downstream scaling law. Step 1 (compute→token power law) is reproducible with a small-model IsoFLOPs sweep. Step 2 — mapping normalized NLL to task accuracy — requires a family of higher-compute anchor models spanning the loss regime (here, the entire Llama 2 herd). Without those anchors the sigmoid is unconstrained and the four-order-of-magnitude extrapolation collapses. Everyone can copy the dense-Transformer-plus-DPO recipe; almost nobody has the prior generation of frontier models to calibrate the forecast that de-risks a nine-figure training run.

关键实现细节 (easy-to-miss tricks).

1. DPO formatting-token masking — header/termination tokens appear in both chosen and rejected, so an unmasked DPO loss issues a contradictory push/pull and produces tail-repetition / premature EOS. Masking them (plus 0.2·NLL on chosen to stop logprob collapse) is what actually stabilizes DPO at 405B.
2. 0.1% long-context data suffices — mixing just 0.1% synthetic long-context samples into short-context SFT jointly optimizes short- and long-context benchmarks; more hurts short-context. DPO can stay short-context entirely (fewer optimizer steps) without regressing 128K performance.