A posture note before the numbers, the same one that opens every note in this series: this is not an NVIDIA collaboration and nothing here is their endorsement. We used the Nemotron model exactly as publicly released, froze it, and measured before and after under identical conditions. We publish because the result says something we think matters to anyone who ships or deploys a foundation model — that a frozen model checking its own work against real tests is a real lever on quality.
GoM wasn't built for this
An honest bit of provenance, because it shapes how to read the result. GoM was not designed to compose datacenter flagships. The architecture behind this series was built for the edge — for a model that learns while you use it, verifies its own work, and improves on-device without a data center in the loop (Notes 1–6). Wrapping a frozen 550-billion-parameter flagship was never the plan.
What changed our minds was watching Inkling — the strong frontier coding result published mid-July — clear a bar that frozen open weights weren't reaching in the setups we track. It made us curious: would the verify-and-select discipline we rely on at the edge transfer, unchanged, to a flagship two orders of magnitude larger than anything we run on a phone? We ran the experiment off-label — but not carelessly: the same pre-specified bars, the same do-no-harm posture as everything else in the series. It transferred — by less than we first thought, and with better evidence than we first had.
Eighteen fixed, three broken — on 429 paired problems
SWE-bench Verified is the human-validated slice of SWE-bench: each task is an actual GitHub issue whose fix is graded by running the project's real test suite. The definitive measurement is paired. Every one of the 429 instances is scored twice — once with the frozen model answering single-shot, the way it is actually deployed, and once with the same frozen model inside the loop — and both arms were graded end to end in a single environment, on one box, on one day. That last detail is not housekeeping; it is the whole reason this number differs from the one this note used to carry.
| Configuration | SWE-bench Verified (n=429 · paired · exact McNemar p = 0.0015) |
|---|---|
| Frozen Nemotron Ultra (550B, deploy precision), answering single-shot | 56.41% (242/429) |
| Frozen Nemotron Ultra (550B) inside GoM's verify-and-select loop | 59.91% (257/429) · +3.50 · 18 fixed, 3 broken |
| The same resolved counts read over the full 500-problem set | 48.4% → 51.4% (+3.0) |
The direction is the part worth sitting with, and it is not unanimous. Twenty-one of the 429 problems came out differently in the two arms. Eighteen fell the loop's way — a bug the bare model missed and the loop fixed. Three fell the other way: the bare model had them, and the loop handed in a patch that failed. We checked whether those three were noise, because earlier runs of this experiment had reported none. All twenty-one discordant instances were regraded three independent times, both arms' patches, 126 gradings in all; every round reproduced 18 out of 18 and 3 out of 3, with zero errors. The three are real, and they have names: django__django-11532, django__django-12273, django__django-16315. Eighteen fixes against three breaks is a strongly favorable, honest trade — and it is what makes this number evidence rather than arithmetic.
How we measured it
Three things make the delta above hard to fake, and they are the details we would want to see if we were reading this from the outside.
Non-leaking by construction. When the loop selects among candidate fixes, it is only ever allowed to see tests it can run for itself — reproduction tests it generates from the issue. The hidden acceptance tests that decide the score are never shown to the model or the selector; they are executed solely by the official, independent SWE-bench grader. That is what keeps the measurement honest, and it is also exactly why three tasks broke: the screen the loop selects on is a proxy for the official label, not the label itself, so it can prefer a patch that passes our test and fails the grader's. Three times in 429, it did. A selector that could never be wrong would be a selector that had seen the exam.
A single-shot baseline, scored in pairs. The "before" arm is the frozen model answering once — the configuration anyone actually deploys — and the "after" arm is the same frozen model drawing candidates and selecting among them. Each instance is scored in both arms, and the comparison is made instance by instance, which is what the exact McNemar test reads: not two percentages, but the twenty-one problems that changed hands. Because the comparison is against the model's own first trajectory, with a fallback behind the selection, "fixed" and "broken" are literally true of individual problems — which is why we can report 18 and 3 rather than two independent scores. The honest cost of that structure is inference: the loop spends the extra attempts, and we make no claim here that it makes a query cheaper.
One environment, one day. Both arms were regraded end to end on 13 August 2026 on one box, at one harness version — SWE-bench 4.0.4, eight workers, newline-terminated patches — with the per-arm resolved-ID sets hashed. No mixing of grading vintages, which is precisely what went wrong the first time. Zero build timeouts across roughly 840 instance gradings. Two instances still fail to grade cleanly, psf__requests-1142 and sphinx-doc__sphinx-9320, and they are the same two in both arms — symmetric, so no arm was advantaged by the infrastructure.
We also measured the ceiling of the full candidate pool, not just the selected fix, so we know exactly where the remaining headroom lives. The current frontier is repairing the near-misses — candidates that are one fix away from passing — which is the lever we are sharpening next. Stated plainly on the other side of that headroom: Ultra with GoM, at 59.91% on the paired set and 51.4% read over the full 500, is a real gain, but it does not clear Inkling's 77.6% — the result that got us curious in the first place. On any reading, we do not claim to clear it. This is a way to make an already-shipped frozen model measurably better at code — not a claim of a new state of the art.
The same thesis, a second benchmark
The loop isn't specific to code. We ran the identical diagnosis on ARC-AGI-2, the abstract-reasoning set from Note 13. Raw, at a matched compute budget, the same frozen flagship solves 13 of 120 on our development set. With GoM, it solves 30. The failure mode mirrors bug fixing — fitting the worked examples is a necessary but weak check — and improving the loop moved the score from 22 to 30 in a single day — six of the eight new solves from selection-side improvements alone, at negligible added compute and no new model calls; two from small, bar-gated repair rounds — all under a regression guard frozen before the run and a separate 20-task holdout left untouched. A staged plan — accept bars and kill criteria written down first — targets further gains.
Why this matters to the people who ship models
The gain above makes a shipped model better at being itself — no training run, no proprietary data, no touched weight. It also turns quality into a dial: draw more candidate attempts and you trade inference for accuracy, elastically, and because the attempts run in parallel the wall-clock stays close to a single pass. The price of the dial is the inference those extra attempts consume; Note 19 measures that cost on this same run. And because the loop runs at deploy precision on commodity silicon — the result is measured on the model as you would actually ship it — it decouples added capability from training capex: a quality-per-watt hedge as data centers drive roughly half of U.S. electricity-demand growth through 2030 (LBNL/DOE 2024; IEA Electricity 2026). The honest framing is the one in the callout above — composition buys quality per training dollar, not a cheaper query. And the same wrap-a-frozen-model idea is the one we run on-device (Notes 4 and 6): capability that upgrades at the edge is capability that doesn't queue for a data center.
What we're building next
Everything above came from GoM's edge architecture used off-label — the verify-and-select discipline built for a phone, pointed at a flagship. The obvious next step is to stop borrowing. We are now building a new GoM architecture designed specifically for this composition role: learned components that judge which candidate fix is actually correct, repair the near-misses that sit one edit from passing, and decide where to spend the compute budget across attempts. A better judge is not a nicety here — the three broken tasks are three cases where the proxy screen preferred the wrong patch, and that is exactly what a trained judge is for. Where this note's loop leans on general-purpose verification, the new design trains those judgments for the outer loop itself. We're testing it on the Blackwell B300 systems this series runs its flagship work on — and the first result is in. Overnight we pointed the loop at formal mathematics: miniF2F, the standard benchmark of formalized competition problems (AMC, AIME, IMO) where an answer isn't a number a judge accepts but a proof the Lean proof checker machine-verifies. On its 244-problem test split, the same frozen Ultra: alone, 44.3%. With the loop selecting among five Lean-checked attempts, 58.6%. And because a proof checker tells you exactly what remains unproven, repair — which has no reliable truth signal on code — comes alive here: one repair pass driven by the checker's own error messages lifts it to 60.2%. That is +16 points on the strictest verification there is, against a bar of +5 fixed before the run — reported, as always, either way. (Self-run, public benchmark; specialized provers score higher — the claim is the delta, not a crown.)
The industry's reflex for "more coding capability" is "train a bigger coder." This is the other answer, measured on real merged bug fixes on a frozen 550B: wrap what's already trained in verification it can't give itself. It helped eighteen times, it hurt three times, and it touched no weight. The definitive run: n = 429, +3.50 points, exact p = 0.0015 — smaller than every figure this note previously carried, and the first one we can defend line by line.
Methodology: all figures are from real, logged runs; every claim traces to an entry in the campaign audit ledger. The SWE-bench Verified figures come from the definitive uniform regrade of 2026-08-13 — n=429 paired instances, both arms graded end to end in one environment (swebench 4.0.4, workers=8, newline-terminated patches): 56.41% → 59.91% (+3.50pp), 18 fixed / 3 broken, exact McNemar p = 0.0015; on the all-500 basis, which counts every instance outside the paired set as unsolved in both arms, 48.4% → 51.4% (+3.0pp). All 21 discordant instances were regraded three times each (126 gradings) and reproduced in every round. Two instances carry residual grading errors, identical in both arms, and are reported as such. The pre-registered bar was +5 points and p < 0.05: significance met, magnitude not met. This supersedes every SWE-bench figure this note previously carried — the Ultra +8.0 at n=50, the 120B Super +6.0 at n=50, and the pooled Super n=149 +6.7 at p = 0.002 — along with the +6.99pp / +6.0pp our frozen headline gave this same run under the 2026-07-26 grading vintage; none of those should be cited. ARC-AGI-2 figures are self-run on the public development set (13 → 30 / 120), under a frozen regression guard and an untouched 20-task holdout, and are labeled internal, not ARC-verified. No result in this note is extrapolated beyond the configurations tested.
Benchmark & attribution: SWE-bench and the human-validated SWE-bench Verified subset — Jimenez et al., “SWE-bench: Can Language Models Resolve Real-World GitHub Issues?” (2023), MIT-licensed. Results in this note were produced with the official SWE-bench evaluation harness and official evaluation images (swebench 4.0.4), self-reported on 429 paired instances of the 500-instance set. Nemotron model names are NVIDIA's; the measurements are ours.