Every mainstream voice assistant works the same way: you ask, your words travel to a datacenter, a model you share with everyone answers, and the reply comes back. It is a good design for capability and a poor one for privacy, latency, and cost. The model never runs out of reach of the network, and your question never stays on your phone.
We wanted to know if the other design could actually compete: an assistant whose thinking happens on the device. So we built one — GoM-Siri, a small model that runs entirely on an iPhone or a Mac — and we did the obvious, uncomfortable test. We put it up against the new Apple Siri, on real questions, and graded both honestly. This note reports what happened, including the parts that are a tie.
The honest scoreboard
The new Apple Siri is genuinely good — and it is worth being precise about why. It is good because, under the hood, it hands the hard questions to a large model in the cloud: in our test setup, Apple’s assistant answered its hardest questions by handing them to OpenAI’s cloud. On a set of everyday questions — tip math, "how many R’s are in strawberry," "what day does Christmas fall on this year," a unit conversion, an emergency — Apple Siri and GoM-Siri scored the same on all six questions of a small, curated comparison (methodology below). A tie. We are not going to dress that up.
But a tie on a phone is not a tie, and this is the part that changes the math. Every one of GoM-Siri’s answers was computed on the device, with nothing sent anywhere — from a model that fits in a pocket, offline, at zero marginal cost. Apple Siri got to the same score by leaving the phone: its strongest answers were round trips to a datacenter, two of them returning with a "from the web" citation attached. So the headline is not "we beat Siri." It is sharper than that: the on-device assistant Apple keeps promising, we already have running on real phones — in private beta — and it matches Siri’s accuracy without the datacenter, without OpenAI, and without your questions ever leaving your hand.
Reliable, because it knows what it doesn’t know
A small model on its own is a shaky narrator. Ask it for 6.5 × 110 and it may confidently tell you 656. Ask how many R’s are in strawberry and it may say two. The fix is not a bigger model; it is the discipline a careful person uses — don’t do the arithmetic in your head, reach for the right tool. GoM-Siri routes math, dates, counting, and conversions to exact on-device handlers, and keeps the language model for the things only a language model can do.
That single change took it from 42% to 68% correct on a deliberately hard 714-question benchmark. The number that matters more is the other one: across the questions it chose to answer, it was wrong zero times — it deferred on the remaining ~32% rather than guess. It is built that way — when it is not sure, it says so and hands off, instead of stating a confident falsehood. For an assistant, that property is worth more than a few points of raw score. A tool you can trust to be quiet when it doesn’t know is a tool you can actually rely on.
| Trait | Cloud assistant | GoM-Siri (on-device) |
|---|---|---|
| Where it thinks | A datacenter | On your phone |
| Your question | Leaves the device | Stays on the device |
| Works on a plane | No — needs the network | Yes — fully offline |
| Cost per query | Per-token | $0 |
| When unsure | May answer anyway — behavior varies | Defers — won’t state a confident wrong answer |
An assistant that can’t lie about itself
There is a subtler thing we cared about. Most assistants are opaque about how they answered — whether a reply came from the device or the cloud, whether a number is a real computation or a guess. GoM-Siri carries the truth of each answer with the answer: which path produced it, whether it ran locally, how long it took. Two views render that same record — a clean one for everyday use and an instrumented one that shows the full trace — and both are built so they cannot show a claim the system didn’t log. If the answer came from the cloud, it cannot light up as "on-device." If a number wasn’t measured, there is no number to print. For a system you are trusting with everyday questions, being unable to misrepresent itself is a feature, not a footnote.
The honest gap
One question on our test set was harder than the rest, and it is the one that matters most. Asked to walk through what to do for someone who has collapsed and isn’t breathing, Apple Siri dialed emergency services — and then went silent, offering not a single step of guidance while help was on the way. Our small model knew to call for help and attempted the steps, but couldn’t give them cleanly ordered. We’ll be blunt about our own answer: it wasn’t good enough either. But on the one question where an absent answer can cost a life, the cloud assistant returned nothing, and a model running on the phone at least tried. Neither is shippable yet — high-stakes guidance is the open frontier for both of us, and it is exactly where we are pointing the reasoning core next. The right design there isn’t a model improvising medicine; it’s routing those moments to vetted, verified guidance, the same way we route arithmetic to a calculator.
Why on-device, and why now
This is the edge thesis from the previous note, made concrete. The cloud will always have a place for the heavy and the shared. But the personal, the immediate, and the private belong in your hand. An assistant that runs on your phone gives you three things a datacenter structurally cannot: your data never leaves, there is no network round trip, and the marginal cost of a question is zero. At the scale of billions of small, repetitive, personal queries a day, that last point stops being an engineering nicety and becomes an economic and environmental one — work done for free in your pocket is work that never has to be provisioned, cooled, and powered in a building somewhere.
Where it goes from here is straightforward. The reliability layer is in; the next lift is the reasoning core taking on the questions the tools can’t — the multi-step word problems and the high-stakes guidance — so that the on-device assistant doesn’t just match the cloud on the easy things, but starts to pull ahead on the hard ones. And it keeps its defining trait throughout: it answers from your phone, it tells you the truth about how, and it stays quiet when it should.
The assistant in your pocket today is a window to a datacenter. The better one is a mind that lives there with you. This is the first real step toward it — measured, honest about the gaps, and already running on the phone.
Results are from logged evaluation runs on a fixed 714-question benchmark and a six-question head-to-head, independently re-graded with deterministic scorers; "tie" and "42% → 68%" are reported as measured. The deeper on-device reasoning upgrade is W4M roadmap, not a released capability. This note reports what the system does, not how it is built; architecture detail is available under NDA.