Training Verifiers to Solve Math Word Problems — L2 #

1. TL;DR #

Autoregressive generation scales terribly on multi-step grade-school math. Instead of a bigger generator, train a separate verifier that scores 100 sampled solutions and return the best. On the new GSM8K benchmark, 6B verification ≈ finetuned 175B — a ~30× parameter saving, scaling better with data.

2. Q1 / Q2 / Q3 #

Q1 — 痛点 (the pain) #

Large language models fail catastrophically at multi-step math reasoning. The root cause is structural, not just capacity: an autoregressive generator has no mechanism to correct its own errors, so a solution that "veers off-course quickly becomes unrecoverable" and a single wrong step dooms the whole answer. Because each step is a fresh failure opportunity, accuracy decays with reasoning depth, and pure generation scales so poorly that a naive log-linear extrapolation implies a $10^{16}$-parameter model would be needed to hit 80% on GSM8K — an absurd figure that dramatizes the scaling problem. A secondary obstacle is the absence of a good benchmark: prior math datasets are either tiny, templatized, low-quality, or lack natural-language solutions.

Q2 — 方法 (the method) #

Two ingredients. (1) GSM8K: 8.5K human-authored grade-school word problems (7.5K train / 1K test), 2–8 steps each, with natural-language solutions and <<...>> calculator annotations, built for high diversity + high quality + moderate difficulty. (2) Verification: finetune a generator for only 2 epochs, sample 100 completions per training problem, label each purely by whether its final answer is correct, and train a separate verifier to output a per-token correctness probability (a token-level value function) with a joint language-modeling auxiliary objective. At test time, sample 100 completions, rank by verifier score, return the top-1 (or take a majority vote among top-k).

核心技术壁垒: the non-obvious insight is that the generator producing verifier training data should be deliberately under-trained (2 epochs), because solution diversity/coverage (test@100) collapses after a few epochs even as greedy accuracy (test@1) keeps climbing. The verifier's entire value comes from ranking a diverse candidate pool; a "better" (more-trained) generator yields a worse verifier. Getting this coverage-vs-accuracy trade-off right is the hardest part to rediscover.

Q3 — 结果 (the result) #

On the full training set, 6B verification slightly outperforms a finetuned 175B generator — a boost "approximately equivalent to a 30× model size increase" — and verification's advantage widens with more data, whereas the finetuning baseline scales weakly. Token-level verifiers beat solution-level ones and overfit less; residual dropout helps both methods; and removing intermediate natural-language steps collapses 6B finetuning from 20.6% → 5.2%.

3. 架构 / 方法图 #

Paper's Figure 4, verbatim (caption: "A diagram of the verification training pipeline."). The pipeline is three stages: finetune a generator (2 epochs) → sample 100 completions per training problem and label each correct/incorrect by final answer → train the verifier for 1 epoch on that labeled pool. The key structural choice visible here is that generation and verification are decoupled into separate networks so the generator is not over-trained.

The training data that drives this pipeline comes from GSM8K itself:

Paper's Figure 1, verbatim (caption: "Three example problems from GSM8K. Calculation annotations are highlighted in red."). Each problem has a natural-language solution with inline <<...>> calculator markup (red). Reader should notice solutions are prose with embedded arithmetic — richer than the pure-equation format of prior datasets, which is what enables interpretable token-level verification later.

The verifier's scalar head sits on top of an otherwise ordinary language model:

The scalar head is implemented as a single bias + single gain applied to one reserved special-token logit; the remaining logits keep serving the joint LM objective. This minimalism is why the verifier can be the same size as (or smaller than) the generator.

4. 作者证明 #

无形式化作者证明 — 仅实证. The paper contains zero numbered display equations and no convergence/variance theorem. It is an empirical study; guarantees are purely experimental. Below is the notation/objective table plus the physical meaning of the (informal) objectives, followed by the 6 minimum sanity checks.

Objectives (physical meaning). Generator: minimize cross-entropy $\mathcal{L}_{\text{gen}} = -\sum_t \log p(s_t \mid x, s_{plus the same LM cross-entropy, added unweighted. The per-token value target means the verifier learns "is this partial trajectory on track to be correct," i.e. a value function over reasoning prefixes — physically, it converts a hard end-of-sequence judgment into a dense signal that "encourages the model to judge the reasoning throughout solutions, rather than merely memorizing the correct final answer."

Loss decomposition (2 terms). The verifier loss = (i) correctness-prediction term, which does the actual ranking work; (ii) language-modeling term, which keeps the verifier fluent in the generator's output distribution — ablated in Fig 6(b) as a "strict improvement."

6 minimum checks:

1. Units/scale of headline claim: "30× model-size boost" = 175B / 6B ≈ 29×, consistent with the 6B-verifier ≈ 175B-finetuning result.
2. Label validity: labeling by final answer alone admits false positives (right answer, wrong reasoning) — the authors acknowledge this explicitly (Fig 13 row 5), so the training signal is noisy, not clean.
3. Coverage logic: using a 2-epoch generator is justified because test@100 peaks in the first few epochs (Fig 3), so the claim "diversity collapses after 2 epochs" is internally consistent with the recipe.
4. Search monotonicity limit: performance is not monotone in $N$ — it rises to ~400 completions then falls, consistent with adversarial over-optimization (Fig 7a), so "more search = better" is bounded.
5. Ablation directionality: large-generator/small-verifier > small-generator/large-verifier (Fig 6c) implies verification is cheaper on the verifier side — consistent with the "coarse heuristics" hypothesis.
6. Underestimate disclosure: reported numbers are a slight underestimate (<1%) due to a calculator bug; fixing it adds ~1% on full-set verification — magnitude too small to affect any qualitative claim.

What a formal guarantee would have wanted: a statement bounding verifier top-1 accuracy in terms of generator coverage (test@$N$) and verifier ranking error, showing when search provably beats the greedy generator — absent here, left entirely to curves.

5. 实验与数据 #

The headline comparison is finetuning vs verification across model + data scale:



Paper's Figure 5, verbatim (caption: "A comparison between finetuning and verification using 6B and 175B model sizes. Verification considers 100 solutions per problem..."). The load-bearing plot: at large training-set sizes verification (both sizes) pulls clearly above finetuning, and the 6B verifier curve meets/exceeds the 175B finetuning curve — the 30× claim. Notice verification underperforms finetuning at small data sizes (overfitting to the answer beats learning reasoning), and 175B verifiers "take off" earlier than 6B.

The baseline's scaling and its coverage-collapse pathology motivate the recipe:



Paper's Figure 2, verbatim (caption: "Final test performance for various GPT-3 model sizes after finetuning on training sets of different sizes..."). Larger models and more data both help, but the trend is weak enough that extrapolation demands absurd scale ($10^{16}$ params for 80%) — this is the quantitative case against pure generation.



Paper's Figure 3, verbatim (caption: "Test solve rate after finetuning a 6B model on the full GSM8K training set, when the model is allowed to make 1 guess (left) or 100 guesses (right)."). The critical observation: test@1 improves monotonically while test@100 (coverage) peaks early then degrades as the model grows overconfident. This single plot is why the generator for verifier data is stopped at 2 epochs.

Regularization sweep shows dropout is a broad win:



Paper's Figure 8, verbatim (caption: "6B finetuning and verification dropout ablations. (a) Finetuning (b) Solution-level verifiers (c) Token-level verifiers"). 20% residual dropout strongly helps finetuning (a) and solution-level verifiers (b) — enough that dropout-regularized solution-level verifiers reach token-level quality — while token-level verifiers (c), already overfit-resistant, gain only slightly.

Qualitative interpretability of the token-level verifier:



Paper's Figure 13, verbatim (caption: "Five cherry-picked samples... A green background color indicates a high verifier score, and a red background color indicates a low one."). Green/red per-token coloring shows the verifier gaining confidence as a correct solution progresses (row 1) and dropping confidence right after a concrete arithmetic mistake (row 4) — direct evidence the scalar head behaves like a value function, not a memorized answer-matcher.

The full hyperparameter table (reproducibility):



Paper's Table 1, verbatim. Key numbers: batch $3.2\times10^{4}$ tokens, max sample length 400, generator 2 epochs / verifier 1 epoch, verifier temperature $0.7$, verifier loss MSE, 100 completions per problem at train and test.

6. 论证链 #

#	Step (paper-internal)	Support
1	Pure autoregressive generation scales poorly on GSM8K (extrapolation → $10^{16}$ params for 80%).	Fig 2 finetuning curves
2	More training epochs improve greedy accuracy (test@1) but collapse candidate diversity (test@100).	Fig 3 left vs right panel
3	Therefore, to get a diverse candidate pool for a ranker, use a lightly-trained (2-epoch) generator to sample 100 completions per problem.	§4.2 recipe; follows from step 2
4	Train a token-level verifier (with joint LM objective) on those completions labeled by final-answer correctness.	Fig 4 pipeline; Fig 6(a)(b) ablations
5	At test time, rank 100 samples and return top-1 (or top-k vote); this beats finetuning at large data and matches a 30× larger model.	Fig 5; Fig 7 test-time compute
6	Verification is dominated by generator quality/coverage (large gen + small verifier wins), and both methods benefit from dropout.	Fig 6(c); Fig 8

7. 实现 cross-reference #

Official code released by the authors: https://github.com/openai/grade-school-math (dataset + calculator sampling). Model training code (GPT-3 finetuning + verifier heads) is [实现未公开] — the 175B/6B GPT-3 checkpoints and internal training stack are proprietary.

核心技术壁垒 (dedicated paragraph). The single hardest-to-replicate insight is the coverage-over-accuracy generator choice. A naive reproduction trains the generator to convergence to maximize its own accuracy, then samples from it for verifier data — and gets a worse verifier, because a converged generator is overconfident and produces low-diversity samples (Fig 3, test@100 collapse). The paper's counterintuitive move is to freeze the generator at 2 epochs specifically to preserve solution-space coverage for the downstream ranker. This decouples "generator quality" from "verifier-data quality," and missing it silently caps verification performance regardless of verifier size or objective.

关键实现细节 (easy-to-miss tricks):

1. The verifier scalar head is not a new linear layer — it is a single bias + single gain parameter shifting/scaling one reserved special-token logit at the LM's existing unembedding (App E), so other logits keep serving the joint LM loss for free.
2. Question-token masking + 100× language upsampling: the joint objective masks loss on question tokens (train only on solution tokens, Fig 12), and because 100 completions are drawn per example, an "equal mix" of LM vs verifier data effectively upsamples the original language data by 100× — a data-balancing subtlety that shapes the LM auxiliary signal.