A 24B model with GoM outperforms a 675B model on legal work.

The setup: nothing trained, everything pre-registered

The models are Devstral Small 2 24B, DeepSeek-V4-Flash, Qwen3-235B-A22B, GLM-5.3-Flash and Mistral Large 3, each served on our own rented GPUs with weights untouched. GoM sits beside the model as a gateway. For LegalBench it does two plain things: it retrieves worked examples from each task’s own training split and shows them to the model, and in one variant it samples five answers and abstains when fewer than three agree. No fine-tuning, no weights touched.

The benchmark is LegalBench, 154 scorable classification tasks. Our first pass used the first 30 items of each task and produced a number we had to withdraw: those files list positive examples first, so the slice was 88% one label and every error the additions made was a false negative. We re-cut a stratified, seeded, 50/50 slice of 30 items per task, froze it with a checksum, and pre-registered the arms and the bar before any model saw it. Scoring is penalized: +1 right, 0 abstain, −1 wrong.

What the bounds actually support

Three sentences carry the weight. GoM beats each model’s own zero-shot answer on every model in every round, nine of nine, with lower bounds that never approach zero. Against the benchmark’s own curated prompt, GoM wins on DeepSeek-V4-Flash in both rounds, ties on GLM and Qwen, and loses on the two Mistral models. And in the pre-registered cross-model contrast, the 24B with GoM beats the 675B running bare by 15.0 points (lower bound +10.1), while the 24B running bare beats it by only 3.3 points. The engine alone does not explain the result; what GoM adds does.

Where it did not work, and what that taught us

Harvey’s LAB is an agentic suite: fifty legal drafting tasks where the model works in a sandbox for up to sixty turns and a rubric judge scores the deliverable. We pre-registered three arms on GLM-5.3-Flash. The first two tried to recover runs that die mid-task; both missed their bar, and the first was worse than a miss: we had written that the recovery layer “rescued five dead runs” before checking its own event ledger, which showed it had fired on none of them. That sentence was withdrawn and is logged as a correction.

The third arm was the one we had real hope for: generate three drafts, screen them on checkable properties a lawyer would recognise (required sections present, citations that resolve, facts that trace to the source documents, defined terms actually used), and select. It cleared its pre-registered bar, +4.1 points with a lower bound of +1.4. Then the decomposition: on the 44 tasks where all three drafts came out intact, selection was worth +0.1 points. All of the gain came from five tasks where one draft collapsed, and a free rule with no verifier at all, skip any draft whose run hit the turn cap, scored higher than the whole screen. On those tasks the judge scores the three drafts almost identically. The drafts themselves are not alike: measured as text they share about a tenth of their wording. What is alike is the score, because intact drafts sit at the judge’s ceiling, and a selector cannot help when the judge cannot tell the candidates apart.

So we changed the one thing we could. A fourth arm, pre-registered before its first call, gives each draft a different approach rather than a different random seed: one drafts from an outline, one drafts from a fact table built from the documents, one is the control. It asks whether a different approach is something the judge can see. It cleared its bar too, +3.75 points with a lower bound of +1.06, and for the same reason: five collapsed drafts avoided. On the 44 tasks where all three drafts survived, the judge saw no difference (−0.2 points). The one thing the directives changed was how often a run fell apart: the outline-first draft collapsed on none of its 49 tasks against four for the control. At this size that is a hint, not a result.

Then we ran it three hundred times

Fifty tasks was a pilot. The weekend’s last campaign re-ran the question at full power: three hundred stratified assignments from the same suite, three drafts each, nine hundred runs, pre-registered before the first call. The screen picked the submitted draft; the bar was the model’s own average across its three drafts. The selected draft beat that average by +7.4 points (task-cluster lower bound +5.6). A free heuristic (skip any draft that ran out of turns) managed +1.6. The screen captured seventy percent of what a perfect selector could have taken, and it never reads a grading rubric: required sections, resolvable citations, grounded facts, nothing a lawyer would call cheating.

The same exam, run on the frozen 24-billion-parameter model from the parity result, cleared its own pre-registered bar at +5.9 points (lower bound +3.6), and there the free heuristic did nothing at all. Two engines, two clears, one mechanism. Honesty requires the other two findings: the pilot’s hint that outline-first drafting prevents collapses did not survive the larger sample and is retracted in our corrections log, and on tasks where all three of the small model’s drafts were sound, the screen slightly hurt: the confidence interval excludes zero on the harmful side. Selection earns its keep where drafts fail, which on hard agentic work is often.

The judge we did not expect to find

Harvey’s protocol scores each deliverable with two API judges, claude-sonnet-4-6 and gpt-5.5, the latter via OpenRouter as served and provider-pinned. That was the only recurring API cost in the programme. We re-judged the same runs with Mistral Large 3 served on our own pod and recomputed the pre-registered verdict both ways: the paid judges said +1.97 points, lower bound −6.6, a miss; the self-served judge said +1.44, lower bound −7.1, a miss. Half a point of difference in the effect, the same decision. Per task the free judge is about four points more lenient with a tight spread, and it ranks runs the way the paid pair does. The leniency is not constant across model families, so it is only sound within one model’s own arm-versus-baseline pair, which is the only way we use it. Every arm after that calibration was judged at zero API cost.

What this does not show

Three limits. It does not show GoM beating a curated few-shot prompt in general; it shows that on one of five models, twice, and on the smallest model the curated prompt is clearly better. The parity result is a specific, pre-registered comparison, a 24B with GoM against a 675B running bare on one balanced slice, not a claim that small models generally replace large ones. It does not show that selection improves legal drafting quality; on the population where quality is all that varies, it showed the opposite, and the diversity arm is the open question. And none of our LAB numbers are comparable to Harvey’s published figures: different judges, different denominators, one model, fifty tasks.

Method & integrity notes

Every arm was pre-registered with its metric and bar frozen in a commit before the first model call; the frozen item lists carry checksums. Baselines are the model’s own average over its samples, never a vote. Rounds are reported per round and never pooled. Twenty corrections were logged during the programme, including the withdrawn slice, the withdrawn rescue claim, and an effect-size call made on eleven tasks that reversed at eighteen. Models were self-served on rented NVIDIA B300 pods (Mistral Large 3 in NVFP4, the rest in FP8 or bf16); no external inference API was used for generation. Receipts for every number in this note live in the GoM repository.

Sources

LegalBench (Stanford) · Harvey LAB · DeepSeek (DeepSeek-V4-Flash) · Alibaba Qwen (Qwen3-235B-A22B) · Z.ai (GLM-5.3-Flash) · Mistral AI (Mistral Large 3) · NVIDIA B300

Qualified partners: request access to the full evidence →