The posture note that opens every note in this series: the numbers below are from real, logged runs; the policy, the bars, and the problem list were sha-frozen before any generation; and negative or missed results appear at the same size as wins. The benchmark is public (SWE-bench Verified); the receipts are reproducible under NDA.
What ran, exactly
Two hundred problems, drawn by seed from the same paired set as Note 15's definitive run. For each problem the frozen flagship generates one candidate fix at a time. After each candidate, an execution-based check — the same non-leaking public-test signal the selection loop already uses — scores it, and the frozen policy either stops (the fix looks done) or samples again, up to five attempts. No human in the loop, no threshold tuned after the fact, no problem excluded.
| Pre-registered bar (frozen before generation) | Result | Verdict |
|---|---|---|
| Quality: resolve ≥ 117/200 (within 1 point of the fixed-budget reference, 119) | 129 / 200 — above the bar and the reference | passed |
| Budget: ≤ 3.25 samples per problem on average | 3.345 | missed |
The anatomy is the interesting part. 73 problems closed on their very first sample — the policy recognized a verified fix immediately and spent nothing more. Eleven more closed on the second, then the tail thinned fast (2, 2, 1). The remaining 111 ran to the cap and kept their best-scoring attempt. Net: the full fixed-budget quality, and then three more points of it, with 33% of the sampling spend saved.
The miss, at full size
The budget bar said 3.25 average samples; the live run spent 3.345. We did the autopsy on the per-sample records rather than accepting our own first explanation — and the first explanation was wrong. The live run's first wave actually matched the replay almost exactly. The real cost hid in two places. First, seventeen problems had a fully passing fix on the very first sample but the independent reproduction step — the extra check the policy requires before it stops — read as failed; a simulation said a single retry on that step would land the run at ~2.95 average samples, comfortably under the bar. Second, the stopping rule's threshold sits in a gap of the score distribution, so the obvious fix (re-tune the threshold) provably changes nothing — the better rule lowers the bar to stop as the remaining budget shrinks, the classic shape from optimal stopping.
Then we measured the first fix instead of trusting its simulation — and the simulation was optimistic. Re-running the reproduction step for every candidate that had lost a close this way recovered only 6 of 74. Most of those failures were the check working, not infrastructure flaking; the retry alone would not have carried the run under the bar. So the correction gets the same size as the claim: the decaying stop-threshold is the fix that goes forward, the retry is a minor assist, and both the original simulation and the measurement that demoted it are in the logged records. We could have shipped this note without the miss — and without the correction of our own autopsy. It would have been a different company's note.
Why this generalizes — early signal from three other model families
The headline generalization first: on a current-generation open model — Google's Gemma-4 26B-A4B — deciding per problem whether to deliberate at all cut reasoning-token spend by 79% — four out of five reasoning tokens simply not spent — with accuracy inside the noise band. When this note was drafted that was a probe-scale figure; the full-set runs landed overnight and it held — same model, same sample count, the cheap mode matches the deliberate one on the full 500-problem MATH set at a fifth of the tokens. The overnight runs also produced the sharpest row yet: on an easier benchmark — GSM8K — the same model is 2.6 points more accurate while spending 86% fewer reasoning tokens — skipping deliberation there isn't a trade, it's a strict win. And the dividend is lawful across the family at matched sample count: deliberation pays clearly on the smallest model (about +21 points under artifact-corrected grading, 61.6 → 82.2 with the same corrected grader on both arms), and nothing at all on the tiers above it — which is exactly why the decision has to be made per problem, not per model. On Alibaba's Qwen3.6-35B-A3B, a frontier open mixture-of-experts model, an agreement-based race matched full-consensus accuracy with 31% fewer reasoning tokens than full-consensus voting (probe-scale, full run scheduled) — a cheaper way to buy the multi-sample quality lift, still roughly 3× a single answer's cost, stated plainly. And on a small edge-class model — Google's Gemma-3 1B — verified selection with a larger in-family judge (Gemma-3 12B) lifted grade-school math (GSM8K) from 60% to just under 80 — most of the gap to a model four times its size — with no training at all.
The deepest of the overnight cuts deserves its own paragraph. Give the deliberating edge model — Gemma-4 E4B — and the dense flagship — Gemma-4 31B — twenty samples each on the full competition-math set (MATH) and majority voting calls it a dead heat — 85.0 against 85.2. But the samples themselves disagree: for 91.6% of the problems, at least one of the edge model's twenty samples is correct — for the flagship, 88.2% — and the flagship's ceiling had already flattened while the edge's was still climbing. Voting cannot see that difference. Only a selector that can verify which sample is right can harvest it — and that 3.4-point gap, invisible to consensus, is exactly the machine this series of notes is about.
The series' claim keeps compounding: a frozen model, wrapped well, does more than it does alone — and now it does so while deciding, per problem, how much of your money to spend. Quality above the fixed-budget reference at two-thirds the budget, live, under rules we couldn't retro-fit. And when one of our own bars missed, it went in the note at full size — because the discipline is the product too.
Methodology: all figures from logged runs. The live pilot's policy, thresholds, problem list, and both bars were sha-frozen before any generation; grading used the benchmark's official harness; no problems were excluded and no thresholds were adjusted after freezing. Cross-family mode-routing figures are full-set (n=500) measurements at vendor-recommended inference configurations with standard symbolic-equivalence grading for math; the remaining probe-scale figures are labeled as such and scheduled for full-set paired runs. No model weights were modified anywhere in this note.
Models & sources
The flagship under test: NVIDIA's Nemotron-3-Ultra-550B-A55B (550B-class) (open weights, served frozen on our stack). Benchmark: SWE-bench Verified — the human-validated subset of SWE-bench — run through the open-source OpenHands agent harness at its standard configuration. The cross-family models probed at the end of this note: Google's Gemma-4 26B-A4B, Gemma-4 E4B, and Gemma-4 31B, plus Gemma-3 1B with Gemma-3 12B as judge; and Alibaba's Qwen3.6-35B-A3B. Math benchmarks: MATH (full sets, n=500) and GSM8K. These families get full-set treatment in Note 19.