A strong result at a few million parameters is easy to set aside. Maybe the model memorized. Maybe it got lucky. Maybe it falls apart the moment you make it bigger. The history of “post-transformer” ideas is mostly a history of things that worked small and broke before they reached the scale where they’d matter. So the honest test of a new architecture is not whether it can solve a hard problem once. It is whether it obeys a scaling law. GoM does — and this note is the curve.
The curve fits its own final loss
We trained a 962-million-parameter GoM language model from scratch. As it learned, its loss fell along a clean power law, fit across more than ten thousand logged points from five million to fifty billion training tokens, with an R² of 0.977. In plain terms: as the model reads more data, its error drops in a smooth, predictable line — the same mathematical signature the major labs use to show that an architecture is real rather than a one-off.
One number is worth slowing down for. Fit the curve on only the early part of training — roughly the first 1.5 billion of the 50 billion tokens, about 3% of the run — and extrapolate more than thirty times further out: it predicts the final loss to within about three hundredths of one percent, 2.7605 forecast against 2.7612 actual. A curve that calls its own endpoint from that far away isn’t noise that happened to trend down. It is an architecture following a law.
The same shape holds across sizes
A single run shows one model improving predictably with data. The harder question is whether the architecture keeps its shape as it gets bigger. So we trained a controlled ladder of GoM models — identical recipe, only the model’s size changed — from 30 million to 962 million parameters, a 32× range, and estimated each model’s loss floor: the best error it settles toward. Plotted against model size, those floors fall along a clean power law with exponent β = 0.245 (R² = 0.938). One architecture, every scale, the same law. That exponent sits inside the 0.07–0.35 band that published scaling studies report for frontier architectures (Kaplan et al. 2020; Hoffmann et al. 2022). Read directly: a bigger GoM is predictably better, and the curve tells you how much the next increment of size buys.
Stable where new architectures usually break
Across the entire billion-parameter run there were zero divergences and zero loss-spike rollbacks. It trained as stably at nearly a billion parameters as at thirty million — which, for a non-attention architecture, is the part that usually doesn’t hold. Most novel architectures don’t fail the idea; they fail the engineering somewhere on the way up. This one didn’t.
| The scaling question | Typical new architecture | GoM |
|---|---|---|
| Predictable loss curve | Often noisy or ad hoc | R² 0.977; fits its own endpoint to ~0.03% |
| Holds across sizes | Breaks before it matters | Power law across 32×, β = 0.245 |
| Trains stably at 1B | Divergence, loss spikes | Zero divergences, zero reverts |
Why a curve is the point
AI’s binding constraint has quietly shifted from whether models can do the work to what the work costs — in compute, in power, in dollars. An architecture that earns more capability per parameter, and keeps earning it as it grows, is a different kind of answer to that problem than building another data center. A scaling law is how you show the lever is real. The rest of this series is what that lever buys: a model that keeps learning after training, reasons where larger systems fall off, and runs on the device in your hand.
Methodology: figures are from a logged 962M-parameter training run (~10,000+ logged points, 5M→50B tokens; power-law fit R² 0.977; fitted vs. actual final loss 2.7605 / 2.7612) and a controlled same-recipe ladder from 30M to 962M parameters (32×), with cross-seed reproducibility validated at the 92M scale (N=3). The cross-size capacity law is fit across a five-rung ladder (30M / 77M / 143M / 267M parameters, anchored by the 962M run) at a fixed data-exponent α = 0.227, giving β = 0.245 (R² = 0.938); each rung's loss floor is a fixed-α extrapolation from its own training trajectory. Zero divergences and zero loss-spike reverts across the billion-parameter run. The base is generation-limited at this token budget and is presented as a foundation for trained heads. Core architecture is a linear-time, non-attention core; construction details are under NDA.