A posture note before the numbers, the same one that opens every note in this series: this is not an NVIDIA collaboration and nothing here is their endorsement. We used the Nemotron models exactly as publicly released, froze them, and measured before and after under identical conditions. We publish because the result says something we think matters to anyone who ships or deploys a foundation model — that a frozen model checking its own work against real tests is a real, cheap lever on quality.
GoM wasn't built for this
An honest bit of provenance, because it shapes how to read the result. GoM was not designed to compose datacenter flagships. The architecture behind this series was built for the edge — for a model that learns while you use it, verifies its own work, and improves on-device without a data center in the loop (Notes 1–6). Wrapping a frozen 550-billion-parameter flagship was never the plan.
What changed our minds was watching Inkling — the strong frontier coding result published mid-July — clear a bar that frozen open weights weren't reaching in the setups we track. It made us curious: would the verify-and-select discipline we rely on at the edge transfer, unchanged, to a flagship two orders of magnitude larger than anything we run on a phone? We ran the experiment off-label — but not carelessly: the same pre-specified bars, the same do-no-harm posture as everything else in the series. It transferred.
It works at both scales — zero regressions
SWE-bench Verified is the human-validated slice of SWE-bench: each task is an actual GitHub issue whose fix is graded by running the project's real test suite. We measured a 50-problem matched subset, giving both arms the identical budget — the same number of candidate attempts per problem — so the only thing that differs between "before" and "after" is the loop itself.
| Configuration | SWE-bench Verified (n=50 matched subset · k=5 · paired) |
|---|---|
| Frozen Nemotron Super (120B), answering directly | 46% |
| Frozen Nemotron Super (120B) inside GoM's verify–repair–select loop | 52% (+6.0 · fixed 3, broke 0) |
| Frozen Nemotron Ultra (550B, deploy precision), answering directly | 58% |
| Frozen Nemotron Ultra (550B) inside the same loop | 66% (+8.0 · fixed 4, broke 0 · +4% tokens) |
The direction is the part worth sitting with. Take every problem where the two arms disagreed — seven of them, across both runs — and all seven fell the same way: the loop solved a bug the bare model missed, and never once broke a bug the bare model had already solved. Two model sizes, each at its native precision, one direction of travel. That shape — help present, harm absent — is the signal we care about more than any single percentage.
How we measured it
Three things make the delta above hard to fake, and they are the details we would want to see if we were reading this from the outside.
Non-leaking by construction. When the loop selects among candidate fixes, it is only ever allowed to see the repository's own public tests. The hidden acceptance tests that decide the score are never shown to the model or the selector — they are executed solely by the official, independent SWE-bench grader. The loop cannot peek at its own exam; it can only check its work against the tests any developer would already have.
Matched budget. Both arms draw the same number of candidate attempts per problem. The "before" arm gets exactly as many tries as the "after" arm — the wrapped arm just verifies, repairs, and selects among them instead of guessing. So the measured lift isolates the loop, not extra sampling.
A validated harness. Before trusting any delta, we checked that our setup reproduces the model's known behavior: a separate 15-problem reproduction of the Super baseline scored 53.3%, with a wide confidence interval (Wilson [30.1, 75.2]) that comfortably contains NVIDIA's own published 60.47%. That check ran the standard published-style agentic setup (hence the ~53–60% range), while the paired experiment ran the deliberately conservative matched-budget regime (hence 46%) — the delta, not the absolute, is the measurement. Fifteen problems can't pin a number precisely — that interval is wide, as expected at n=15 — but it confirms the pipeline is faithful, not quietly broken.
We also measured the ceiling of the full candidate pool, not just the selected fix, so we know exactly where the remaining headroom lives. The current frontier is repairing the near-misses — candidates that are one fix away from passing — which is the lever we are sharpening next. Stated plainly on the other side of that headroom: Ultra with GoM, at 66%, is a real gain, but it does not clear Inkling's 77.6% on the full 500-problem set — the result that got us curious in the first place. Our 66% on a 50-problem subset is not directly comparable to a full-500 number, but on any reading we do not claim to clear it. This is a way to make an already-shipped frozen model measurably better at code — not a claim of a new state of the art.
The same thesis, a second benchmark
The loop isn't specific to code. We ran the identical diagnosis on ARC-AGI-2, the abstract-reasoning set from Note 13. Raw, at a matched compute budget, the same frozen flagship solves 13 of 120 on our development set. With GoM, it solves 30. The failure mode mirrors bug fixing — fitting the worked examples is a necessary but weak check — and improving the loop moved the score from 22 to 30 in a single day — six of the eight new solves from selection-side improvements alone, at negligible added compute and no new model calls; two from small, bar-gated repair rounds — all under a regression guard frozen before the run and a separate 20-task holdout left untouched. A staged plan — accept bars and kill criteria written down first — targets further gains.
Why this matters to the people who ship models
Every gain above makes a shipped model better at being itself — no training run, no proprietary data, no touched weight. It also turns quality into a dial: draw more candidate attempts and you trade inference for accuracy, elastically, and because the attempts run in parallel the wall-clock stays close to a single pass. Ultra's +8 points cost roughly 4% more tokens. And because the loop runs at deploy precision on commodity silicon — the Ultra result is measured at native FP4, the format you actually ship — it decouples added capability from training capex: a quality-per-watt hedge as data centers drive roughly half of U.S. electricity-demand growth through 2030 (LBNL/DOE 2024; IEA Electricity 2026). The honest framing is the one in the callout above — composition buys quality per training dollar, not a cheaper query. And the same wrap-a-frozen-model idea is the one we run on-device (Notes 4 and 6): capability that upgrades at the edge is capability that doesn't queue for a data center.
What we're building next
Everything above came from GoM's edge architecture used off-label — the verify-and-select discipline built for a phone, pointed at a flagship. The obvious next step is to stop borrowing. We are now building a new GoM architecture designed specifically for this composition role: learned components that judge which candidate fix is actually correct, repair the near-misses that sit one edit from passing, and decide where to spend the compute budget across attempts. Where this note's loop leans on general-purpose verification, the new design trains those judgments for the outer loop itself. We're testing it on the same Blackwell B300 systems the Ultra result was measured on — and the first result is in. Overnight we pointed the loop at formal mathematics: miniF2F, the standard benchmark of formalized competition problems (AMC, AIME, IMO) where an answer isn't a number a judge accepts but a proof the Lean proof checker machine-verifies. On its 244-problem test split, the same frozen Ultra: alone, 44.3%. With the loop selecting among five Lean-checked attempts, 58.6%. And because a proof checker tells you exactly what remains unproven, repair — which has no reliable truth signal on code — comes alive here: one repair pass driven by the checker's own error messages lifts it to 60.2%. That is +16 points on the strictest verification there is, against a bar of +5 fixed before the run — reported, as always, either way. (Self-run, public benchmark; specialized provers score higher — the claim is the delta, not a crown.)
The industry's reflex for "more coding capability" is "train a bigger coder." This is the other answer, measured on real merged bug fixes at 120B and 550B in the same week: wrap what's already trained in verification it can't give itself. It helped at both scales, it never hurt, and it touched no weight. The powered pooled run has now landed: n = 149, +6.7 points, 10 fixed / 0 broken, p = 0.002 — significance achieved.
Methodology: all figures are from real, logged runs; every claim traces to an entry in the campaign audit ledger. SWE-bench Verified figures use an n=50 matched subset with k=5 candidate attempts in both arms, scored by paired McNemar; both per-scale deltas clear a pre-specified +5-point bar (p = 0.25 / 0.125 at this sample size), and the powered pooled Super run (n = 149, Jul 23) met the bar: +6.7 points, 10 fixed / 0 broken, exact p = 0.002. The 15-problem reproduction (53.3%, Wilson CI [30.1, 75.2]) validates the harness against NVIDIA's published 60.47%. ARC-AGI-2 figures are self-run on the public development set (13 → 30 / 120), under a frozen regression guard and an untouched 20-task holdout, and are labeled internal, not ARC-verified. No result in this note is extrapolated beyond the configurations tested.
Benchmark & attribution: SWE-bench and the human-validated SWE-bench Verified subset — Jimenez et al., “SWE-bench: Can Language Models Resolve Real-World GitHub Issues?” (2023), MIT-licensed. Results in this note were produced with the official SWE-bench evaluation harness and official evaluation images, self-reported on a matched 50-instance subset. Nemotron model names are NVIDIA's; the measurements are ours.