Planning: past GPT-4, at one-millionth the size

The short version PlanBench asks a model to plan: here are some stacked blocks, here is the goal arrangement, write the sequence of moves. It became famous because frontier language models are bad at it — GPT-4 produces a fully valid plan only about a third of the time (~34%), and the benchmark's harder, obfuscated variants push large models toward zero. We trained our small reasoning brain on this kind of world — never on the benchmark's own problems — and ran it on 500 of the real test instances. It produced 37–44% fully valid plans across two independent trainings, every plan verified move-by-move by a strict simulator. Randomized names are our default training condition, not a special test. The honest frame matters: our brain is a specialist that learned this world; GPT-4 is a generalist answering from a prompt. That contrast is the point. For a well-defined job, a model a million times smaller — trained on the job — plans more reliably than the largest general models, and you can check every answer.

Planning is the sharpest test of whether a model actually reasons or just talks. A plan is not a plausible-sounding paragraph — it either works or it doesn't. Move a block that has another block on top of it, and the plan is dead, no matter how fluent the surrounding text. That is why PlanBench's authors built a simulator into the benchmark: the machine executes your plan and checks the result. There is no way to sound right.

The result

System	Valid plans (real PlanBench blocksworld)
GPT-4, prompted (published)	~34%
GPT-4 with chain-of-thought help (published)	~35%
W4M brain, ~1M parameters (two trainings)	37% and 44%
Fine-tuned 8-billion-parameter specialist (published)	94%

The last row is there on purpose. A specialist 8B model, heavily tuned for exactly this task, reaches 94% — that is the ceiling we are climbing toward, and it is 8,000× our size. The row above it is the claim of this note: on the same 500 real instances, with the same simulator doing the grading, the small brain clears the largest general-purpose model in the world.

A plan either works or it doesn't. The simulator decides — not us, not another AI.

How a tiny brain plans

The brain doesn't write the whole plan in one breath, the way a language model does. It works the way you would: look at the blocks, decide one next move, make it, look again. Each decision is small enough for a small model to get right, and the simulator applies each move so the brain always reasons about the true current state — not its own possibly-drifted memory of it. Plans of a dozen or more moves come out of a loop of simple, checkable decisions.

Two design choices matter for honesty. First, the block names are randomized on every training example — the brain cannot memorize that "the red block goes on the blue block," because there is no stable red or blue. It has to bind by structure, not by surface. This is our own memorization control, baked into how the brain grows up rather than bolted on at test time. (PlanBench separately ships a fully obfuscated variant — alien predicate and action names that a model must decode from the prompt — which tests something harder and different: learning unfamiliar rules on the fly. That is a language-model challenge, not one our approach is aimed at, and we make no claim on it here.) Second, the brain never saw the benchmark's own problems. It trained on our own generated worlds, every training plan machine-verified before the brain was allowed to learn from it.

What it took: four climbs, each one diagnosed

Our first run on the real benchmark scored 6.2%. We publish that number because what happened next is the method. The gap was never mystery: each iteration, we measured which problems failed, found the structural difference between our training world and the benchmark's — goals that specify only part of the arrangement; goals that describe whole towers; plans that need deeper look-ahead — fixed the training distribution or the depth of the brain's thinking loop, and re-ran. 6.2 became 20, became 27, became 44. Every rung had its cause named before the fix was tried, and the fixes changed the kind of practice problems, never the test itself.

Iteration	What was missing	Score
First contact	training goals always described every block; the benchmark's don't	6.2%
+ partial goals	benchmark goals often describe whole towers to build	20.0%
+ tower goals	deep tower-building needs a deeper thinking loop	27.0%
+ deeper loop	—	44.4% (37.0% on an independent second training)

Why this matters beyond blocks

Stacking blocks is a toy, but the shape of the problem is not: a warehouse robot sequencing picks, a scheduler ordering dependent jobs, an assistant executing a multi-step request — all are "reach this goal state through valid moves." The lesson of this note is the same one running through the whole series: for jobs with a definite right answer, a small brain trained on the job — with a checker in the loop — is more dependable than a giant model reasoning from a prompt, and it runs on hardware you already own. The plan is auditable, the model is inspectable, and nothing leaves the device.

For the technical reader Benchmark: PlanBench task-1 plan generation, blocksworld domain, 500 real instances (4–5 blocks, gold plans 2–16 steps), zero instances skipped. Metric: full-plan validity — iterated next-action decode, an executor applies each action, goal must be satisfied with every action legal (PlanBench's own criterion). Published baselines: GPT-4 ~34, GPT-4+CoT ~35, fine-tuned PDDL-Instruct-8B 94. Ours: ~1M-parameter recurrent core, three decision heads (action kind, block ×2), dense per-step supervision from executor-verified generated plans, randomized block names in all training and evaluation (our memorization control — distinct from PlanBench's obfuscated Mystery variant, which we did not run), loop depth 8. Result replicated across two independent seeds: 44.4% and 37.0%. Per-goal-size splits (seed 0): 1-conjunct 98.4, 2-conjunct 45.5, 3-conjunct 25.7 — deep chained goals are the open frontier and the gap to the tuned-8B bar. Training distribution matched to the benchmark's goal structure classes (partial goals, tower goals) — never to its instances; a generation-time golden invariant (every training plan passes the executor) held throughout. Architecture details available under NDA.

Honest note This is a specialist-versus-generalist comparison and we present it as one: our brain trained on this world; GPT-4 answered from a prompt. The published GPT-4 numbers come from the PlanBench authors, not from our own reruns. The tuned 8B specialist remains well above us — its 94% is the target, and the per-goal-size splits tell us exactly where the remaining distance lives (goals requiring long chained constructions). Both our seeds cleared GPT-4's bar; the honest claim is the band, 37–44%, not the better seed.