Substitution — the smaller model does the bigger model's job

The short version A production serving stack runs a model once per query. Any efficiency claim has to beat that configuration — not a multi-sample one nobody deploys. Held to that bar, two mechanisms survive with full-set receipts: substitution — serve a smaller model, sampled a few times with selection over the samples, in place of a bigger model run once — and verified selection — where the work is checkable, generate candidates and keep the one that passes the checks. The first frees the GPU capacity difference between the models. The second raises accuracy on a frozen model without breaking anything it already solved. All numbers from logged runs; negative and missed results in this series publish at the same size as wins.

Update, August 2026. The July version of this note measured its savings against multi-sample voting configurations. Our own baseline-control review found that comparison unfit — a production stack runs a model once, and savings measured against a configuration nobody deploys aren't savings. We withdrew the note rather than patch it. Every number below is measured against single-shot. The numbers are smaller than July's. They are also real.

The posture note that opens every note in this series: the numbers below are from real, logged runs at vendor-recommended inference configurations, graded with standard public harnesses — symbolic-equivalence checking for math, execution against held-out tests for code; full-set results are n=500 per cell with zero generation errors; receipts are reproducible under NDA.

What the audit was

In July we asked how much of the inference bill buys nothing, and answered it by comparing sampling configurations against each other. The error was the comparison universe: almost none of those configurations is what an operator actually runs. When we re-anchored every cell to single-shot — one query, one generation, the deployment default everywhere — the claims split cleanly in two. Comparisons between exotic configurations stopped mattering, because the baseline they beat isn't anyone's bill. And one construction kept winning anyway: a smaller model, sampled a few times, with selection over the samples, against a bigger model run once. That construction is this note.

Substitution — the receipts

Same benchmark, full 500-problem competition-math set, same grading, every comparison against the same model family's bigger member run once. The smaller-plus-selection side wins the flagship match and posts consistent lifts across every open family we measured:

Points gained over the same model run once — sampled a few times, selection over the samples

Accuracy lift over the same model run once, per family, full-set cells (n=500, zero generation errors). The lift is what makes substitution work: it closes the gap to the next model tier up, so the smaller model plus selection can take the bigger model's traffic.

The lift itself is not the product — the trade it enables is. On Google's Gemma family, the 4B-active E4B, sampled a few times with selection, scores 84.6 on the full set; the 31B-dense member of the same family, gemma-4-31B, run once, scores 82.4. Each E4B sample touches about a seventh of the active parameters. On OpenAI's gpt-oss family the same construction inverts scale outright: the 20B, composed, beats the 120B — double-seeded with pooled exact p = 0.0019 on the paired head-to-heads, and after a third seed landed on both sides, every 20B seed beats every 120B seed, 9 of 9 pairings, with the direction never flipping. At ~1T scale, on a flagship family we instrument under NDA, the direction holds too (+1.1 points) — but that cell's confidence interval spans zero, so we report it as directional, not proven. Serving stacks are sized for the model they run; when the smaller model plus selection matches the bigger one's answers, the difference is reclaimed GPU capacity — and at fleet scale, megawatts.

The claim names what an operator stops doing: serving the big model for traffic the small one now handles. Nothing about the workload changes — only which model burns the watts.

Verified selection — where the work checks itself

Substitution covers work where answers can't be checked mechanically. Where they can — code with tests, math with verifiers, anything with an executable acceptance criterion — selection gets strictly stronger, because the selector stops guessing. On SWE-bench Verified with an OpenHands agent on NVIDIA's frozen open flagship, the 550B Nemotron Ultra (the same model line as Note 18), we generated candidate patches and kept the first that passed the instance's public tests — the held-out grading tests stay invisible to selection. Paired against single-shot on n=429 instances: 54.3% → 61.3%, exact McNemar p = 1.9×10⁻⁹, for +1.5% total tokens.

SWE-bench Verified, n=429 paired — what changed, instance by instance

The do-no-harm row is the deployable property: on a frozen model, with selection keyed only to public tests, the change is monotone — 30 fixed, 0 broken. That's what lets this run in production rather than in a lab.

Who should care

If you buy inference at scale, the substitution table is the line-item: traffic served by a bigger model that a smaller one plus selection now handles at parity or better. The claim is stated against your actual configuration — the model run once — not against a sampling scheme nobody deploys.

If you sell inference, the same result is capacity: every query moved down a model tier frees the difference in active compute, on hardware you already own, with no weight changes and no retraining.

If you ship models onto devices, the E4B row is the one to read twice: an edge-scale model with selection posting flagship-family answers is the difference between "the device handles it" and "call the datacenter."

If you evaluate models, the audit is the takeaway: a parity or savings claim is only as real as its baseline, and the only baseline that corresponds to a bill is single-shot. We re-based our own series on it; we'd like to see the comparison standardized.

The honest edges

What's measured and what isn't: the per-family lifts and the 84.6-vs-82.4 flagship match are full-set cells (n=500, zero generation errors); the 20B-vs-120B result is double-seeded with the pooled p reported, and the third seed extends the direction, not the test. The SWE-bench result is execution-verified end to end on a frozen model. The ~1T in-family replication is stated without numbers here; its receipts are NDA material. Selection internals — how candidates are ranked and when sampling stops — are proprietary. And this series' standing rule applies twice over now: the last live pilot missed one of its two frozen bars and published the miss at full size, and this note's own July version is withdrawn by name above. The contract doesn't change when the miss is ours.

Methodology: all figures from logged runs at vendor-recommended inference configurations; math graded with standard symbolic-equivalence checking; code and agentic results execution-verified against held-out tests that selection never sees. Full-set cells are n=500, zero generation errors. Every comparison in this note is against single-shot — the same model, or the bigger family member, run once at its recommended configuration. Per-instance records and reproduction paths are available to qualified partners under NDA. No model weights were modified anywhere in this note.

Models & sources

The open-weight families measured in this note, linked to their makers' releases: OpenAI's gpt-oss-20b and gpt-oss-120b; Google's Gemma family (E4B and the 31B-dense gemma-4-31B); Microsoft's Phi-4; Alibaba's Qwen-3.5; Liquid AI's LFM2.5 models; and NVIDIA's Nemotron-3-Ultra-550B-A55B (the frozen 550B flagship of Notes 16–18). All served at vendor-recommended configurations; every claim in this note is from our own logged full-set runs, not vendor-reported numbers. Benchmarks: MATH (full sets, n=500 per cell) and SWE-bench Verified (n=429 paired).