The interesting question about a reasoning model is not whether it can solve a hard problem. It is whether it keeps solving once the problem gets long — once the answer requires holding a plan together over many dependent steps with no room to drift. That is where cost climbs, where confidence and correctness come apart, and where the gap between a good demo and a dependable system shows up. Tower of Hanoi is a clean way to see it: the rules are trivial, but the shortest solution doubles in length with every disk you add. Depth is the whole test.
Frontier reasoners are genuinely good — until they aren’t
We want to be precise and fair here, because the honest version is more useful than the flattering one. On Tower of Hanoi, the strongest frontier reasoning models are good. Under a fair, generous token budget, one solves the 10-disk problem and another solves 9 disks — real long-horizon reasoning, not a fluke.
Then the floor falls out. At 11 disks and beyond, every frontier model we tested drops to 0%. Not "degrades gracefully," not "gets it sometimes" — zero. The horizon gets one notch longer than they were built to carry, and the whole chain comes apart at once. That wall is the point. If your product depends on move 1,800 of a plan, this wall is the number to diligence. The failure is not a knowledge gap you can patch with a better prompt; it is a depth limit baked into how these models hold a plan together.
A 37M model that solves past its training horizon
Now the other side. A GoM with 37 million parameters — small enough to live on a phone — was trained only on Tower of Hanoi problems of 3 to 8 disks. It never saw 9, 10, or 11 during training. Asked to solve them anyway, it generalizes past the wall: it produces correct long-horizon solutions in the exact regime where the frontier models we tested return nothing.
This is generalization, not a trophy number. The model learned the structure of the task from short instances and carried that structure into depths it had never been shown. Retrain it from scratch several times and re-test — an independent reproduction — and the 10-disk case comes back at roughly 80%.
At 11 disks, the headline result. We run several independently trained copies of the model and let them vote on every move, with the ability to undo a bad move and try another — a committee search. (This is a batch evaluation configuration — minutes to a couple of hours per problem at these depths — not the phone-scale single model; the compute is an explicit, bounded budget chosen up front.) Given the complete 100-problem test set, the committee solved 100 of 100 — statistically, a true success rate of at least 96.3% — with 96 of the 100 solutions at exactly the mathematically minimum length, including a perfect 1,791-move trajectory from models that never saw a problem longer than 255 moves in training.
The comparison that explains it: a single model committing to its best guess at each step solves about 50% of the same problems — and pooling all four models’ independent single-track attempts still yields 50%, because the hardest problems defeat every individual model. The committee solves all of them. The knowledge was already in the weights; single-track search was discarding it. Our previous best, averaged over seven seeds, was 79.3%. The moment errors became recoverable and decisions became votes, the failure mode didn’t shrink — it vanished.
Deeper still: at 12 disks — double the solution horizon — the committee went 100 for 100 (true rate ≥ 96.3% again), 94 of 100 exactly optimal, including a perfect 3,464-move trajectory. The same committee also swept the shallower unseen depths — 20/20 at 9 disks and 20/20 at 10 — so every measured rung from 9 through 12 sits at 100% under one configuration.
The 13-disk bank is now fully solved: 20 of 20, seventeen exactly optimal. The last holdout tells the whole story in miniature: it failed two clock-capped attempts at ~1.3 million search expansions each, then solved — at exactly the optimal 6,824 moves — when a third run was allowed 1.36 million. The wall was never ability; it was minutes. At 14 disks the bank is complete: 20 of 20, every single solution exactly optimal — including a 14,455-move trajectory executed perfectly by models whose longest training episode was 255 moves, 57 times deeper than anything they saw in training. The pattern repeated at this depth exactly as at 13: every problem that clocked out at a 4-hour budget solved, at exactly optimal length, when given 10. Six consecutive out-of-training depths now sit at 100%. Every miss along the way was the same kind: the search ran out of budgeted hours, never out of correct moves. At these depths the solve rate is a question of how much thinking time you buy.
What matters is the categorical fact, and it is not close: models you could carry in your pocket are operating in a regime where the best available reasoners score nothing.
| Tower of Hanoi, by horizon | Frontier reasoners (best tested) | GoM (models trained on 3–8 disks) |
|---|---|---|
| N = 9–10 disks | Strong — one solves N=10, another N=9 (fair token budget) | Committee search: 20/20 at both 9 and 10 disks (single-model cross-seed ~80% at 10) |
| N ≥ 11 disks | 0% — all tested models fail | Committee search: 100/100 at both 11 and 12 disks (95% CI ≥ 96.3% each) |
| Run-to-run behavior | Varies between runs | Identical, repeatable answer every run |
Same input, same answer — and it knows when it doesn’t know
There is a property here that matters as much as any benchmark, and it is one frontier models structurally cannot offer: determinism. A GoM exact-executes. Give it the same problem and it does the same thing every time and returns an identical, repeatable answer. Frontier outputs vary run to run — same prompt, different paths, occasionally a different conclusion. For a long-horizon plan that something downstream will act on, "usually right, but I can’t promise the same answer twice" is not a foundation you can build on.
The second half of assurance is restraint. When a GoM is not confident, it declines rather than confabulates — there is a confidence floor below which it would rather say nothing than invent a plausible, wrong answer. In long-horizon settings, a fluent guess is the most expensive failure mode there is, because it is the one that looks like success. A model that can hold a plan to depth, repeat itself exactly, and stop when it is unsure is a model you can actually wire into something.
See it run
Demo Recorded · real logged runsHow far does the depth ladder go? The banks in this note stop at 14 disks only because that is where exhaustive evaluation stops. With a subgoal system on top, the same architecture executed the full 30-disk tower — all 1,073,741,823 moves, exactly optimal, zero errors — in ~9 minutes on a laptop CPU (and a scrambled 30-disk instance, 536M moves, in 139 seconds). That run has a note of its own →
Methodology: all results are real, logged evaluation runs, not live or cherry-picked sessions. Frontier Tower of Hanoi figures are from the best models we tested under a fair token budget (one solves N=10, another N=9; all tested models score 0% at N≥11). The 37M GoM was trained only on 3–8 disk instances; single-model 10-disk cross-seed reproduction is ~80%, and the committee configuration sweeps 9- and 10-disk at 20/20 each (19 and 18 exactly optimal). At 11 disks, a committee of GoM checkpoints guiding a backtracking search solved the complete 100-problem evaluation bank 100/100 (Clopper–Pearson 95% CI [96.3%, 100%]; 96 of 100 exactly optimal — e.g. a 1,791-move problem solved in exactly 1,791 moves), versus ~50% for single-model beam-128 on the same problems — a ceiling the four-seed union of independent beams does not raise — and a 79.3% seven-seed historical average. At 12 disks the committee solved the complete 100-problem bank 100/100 (Clopper–Pearson 95% CI [96.3%, 100%]; 94 of 100 exactly optimal), including a 3,464-move problem solved in exactly 3,464 moves. The 13-disk bank is complete at 20/20, seventeen exactly optimal — both initial misses were wall-clock budget exhaustions and both converted on larger-budget re-runs (5,901 vs 5,887 optimal; 6,824 exactly optimal). Every miss in the campaign has been a budget line, never an incorrect solution. The 14-disk bank is complete at 20 of 20, all exactly optimal — including a 14,455-move problem solved in exactly 14,455 moves (57× the longest training episode). Exact-execution figures (identical repeatable answers; decline-rather-than-confabulate behavior) are properties of the architecture, verified across runs. Per-task eval conventions are reconciled in the data-room paper.