Right once is easy. Right every time is the product.
The identical Tower-of-Hanoi prompt, sent 8 independent times to each system. GoM replays the byte-identical move sequence every run. Frontier models hold together on shallow puzzles — then shatter into different answers and false “DONE” claims as depth grows, even at temperature 0.
W4M Research · July 2026 · 6 frontier models × 3 depths × 2 temperatures × 8 runs
1
distinct move sequence from GoM across every replay — byte-identical, pass^8 = 1.0 by construction
8 / 8
runs of DeepSeek-V4 at 9 disks that each produced a different move sequence — eight answers to one question
8
false “DONE” claims from Claude Opus 4.8-fast in 8 runs at 9 disks — confidently finished, verifiably not
The task — Tower of Hanoi. Three pegs and a stack of disks: move one disk at a time, and never place a larger disk on a smaller one. A perfect solution for N disks takes 2N−1 moves from the canonical start — the optimal 31-move path at 5 disks — so the difficulty doubles with every disk added. The question here is not whether a model can solve it once, but whether 8 identical requests give the same answer.
Pick a model. Raise the disks. Watch the column of matching chips break.
Loading logged runs…
Each row is one real logged run. The colored chip is the run’s trajectory fingerprint — the SHA-256 of its normalized move sequence, shown as its first six characters; same color = same bytes. A column of one repeated chip is determinism you can see. At 5 disks, note the frontier’s temp-0 fingerprint is the same hash as GoM’s — both find the optimal 31-move path, byte-for-byte. Then raise the disks.
Why one-shot benchmarks hide this
pass@1 tells you a system solved it once. Production runs the same job again and again. Drag the slider: a 90%-per-run agent looks fine in a demo and fails silently at scale — the measured frontier pass^8 above is this effect, on a real task.
90%-reliable agent43%
GoM (exact-execute)100%
The full matrix
Every model, every depth, at the temperature selected above. p@1 = fraction of the 8 runs that solved it. p^8 = all 8 runs valid. traj = distinct trajectories in 8 runs. ⚠ = false “DONE” runs.
Honest note
We show the frontier at its best as well as its worst: at 5 disks and temperature 0, most frontier models are perfectly deterministic and correct — several produce the exact optimal-path hash GoM produces. The claim is not that frontier models are always random; it is that their repeatability degrades with problem depth and cannot be guaranteed, while GoM’s exact execution (it always takes its single highest-scoring move — no sampling to vary — and we reloaded the model from disk 8 separate times) is byte-identical by construction — pass^8 = 1.0 at every depth it has a checkpoint for. GoM has no 9-disk checkpoint in this harness, so no GoM number is claimed at N=9; the frontier collapse at N=9 stands on its own. Fingerprints are SHA-256 hashes of normalized move sequences; runs whose output could not be parsed as moves are marked “no parse.” Frontier runs used a generous token budget (up to 32K, scaled with depth) via each vendor’s API. External context: published agent-reliability work reports ~35.6% false-success rates (arXiv 2606.09863) and pass^8 < 25% on τ-bench-style tasks.
Provenance: every number renders live from the logged result files shipped with this page — frontier determinism_data.json (6 models × N∈{5,7,9} × temp∈{0, 0.7} × 8 runs) and GoM gom_determinism_data.json (8 independent re-loads per depth, exact 3-peg verification). Raw transcripts and methodology are available to qualified partners under NDA — request access.
Same input, same answer, every time — that’s an audit trail, not a dice roll.
See the rest of the deck: in-session learning, auditable step-by-step reasoning, verified arithmetic, and the measured economics.