The task being measured
An agent that plans dog walks — and can't fake it
Give it a starting point and a few dogs, each with an address, a walk length, and constraints — a comfortable temperature band, a morning or afternoon window, hills to avoid, medication to administer. A small language model drives the whole thing: it geocodes the addresses, solves the visiting order over real streets, checks the weather for each dog's actual walk window, and returns a schedule with a per-dog safety verdict. It does that by calling four tools and relaying what they return.
Street addresses → coordinates (Nominatim / OpenStreetMap).
Hourly forecast for a place and time window → a deterministic OK / CAUTION / SHORTEN / DO-NOT-WALK verdict.
Elevation around a dog's home → whether the hills exceed that dog's stated tolerance.
Real walking distances (OpenRouteService + OR-Tools) → the visiting order and a timed schedule; returns feasible:false when the windows can't all be met.
The model is the least-trusted part of the system, and the measurement only works because it's boxed in. A referee validates every tool call against its schema before it runs. An auditor — plain code, no AI — refuses a finished plan unless every dog's walk has a real weather check at its own location. And the plan is submitted through a structured submit_plan whose validated arguments are the answer, never parsed from prose. Those checks are what make a "pass" mean something — the deterministic finish line a model can be measured against. They are also why the cheap models hold up: the box is doing work the model never has to. Watch it run live →
The verdict
With the hard calls in code, four of six models are indistinguishable — and 93× apart in price.
Once the route solving, the safety verdicts and the feasibility ruling live in deterministic code, reliability stops separating small models: four of six passed every run. Price separates them completely. The same finished walk plan costs $0.0004 from the cheapest model and $0.033 from the frontier baseline, and that one model spent 75% of this sweep's entire bill. Per thousand plans: 36 cents against $33.
Perfect, fastest of the cheap tier, and the least expensive model in the sweep. Now the site default.
Also flawless — and it alone burned 75% of the whole sweep's cost. The reliability is free; the certainty is not.
Perfect, and it answers faster than anything else in the field — a median plan in 11 seconds, at nearly 1,400 output tokens per second. Speed is its own kind of cheap: the same plan, while a slower model is still thinking. (Still not the default — ling is 3.9× cheaper.)
Reliability
Pass rate, with the doubt drawn in
Five runs can't prove 100%. Each dot is the observed pass rate; the whisker is the 95% Wilson interval — the honest range the true rate could sit in given only 35 samples. Note how wide even a perfect score's floor is, and how far the bottom two models' intervals reach.
Overall pass rate by model
35 runs per model · dot = observed rate, whisker = 95% CI
Cost vs. reliability
The frontier tax
Plot what a plan costs against how often it lands and the trade-off stops being abstract. The upper-left is the prize: reliable and cheap. ling-3.0-flash sits there almost alone. haiku-4.5 matches its reliability two orders of magnitude to the right.
Median cost per plan × pass rate
cost axis is logarithmic · each point is one model
Where they break
One scenario carries most of the difficulty
Averages hide the shape of failure. Out of five runs per cell, four models are solid green everywhere. What difficulty remains sits in one column: grand-tour — six dogs carrying every constraint at once, a comfort band, meds, hills, walk windows and a sunset cutoff, in a single outing. Both slower models lose most of their runs there, and no other column costs anyone more than one run — except single-easy, where qwen3-8b twice talked itself out of walking one dog.
How they fail
Ten runs out of 240, and they are not the same kind of thing
Lumping failures into one number lies. Two were the provider, not the model: a backend that refused six attempts, and one call that never returned. The other eight are the models' own — five livelocks, where a model never satisfied the deterministic auditors before its rounds ran out, and three refusals, where a model declared a perfectly walkable roster impossible.
The two slower models fail in different ways
qwen3-8b is the only model whose accuracy is genuinely behind, and its losses are self-inflicted rather than provider trouble. Three refusals came from arguments it invented — a midnight sunset, morning windows on stops that never asked for one — and three more from livelocks on the heaviest scenario. It also thinks the hardest of any model here, a median of 2,041 reasoning tokens per run, and costs 18× the default.
gpt-oss-120b is accurate and slow. Both of its graded losses were livelocks on that same heaviest scenario; everything else it passed. The cost is time: ~25 output tokens a second and a median plan of 128 seconds, against 15 for the default. It was also the only model to break the tool schemas outright, calling the weather tool 22 times without a latitude — caught by the referee, bounced back, and corrected inside the run.
The one that matters: a model that talks itself out of a walk
Three runs, all from qwen3-8b, declared an easy roster impossible — twice on single-easy, one dog and a thirty-minute walk. The planning wasn't wrong; the arguments were. It told the route tool that sunset fell at 00:00 and stamped a morning window on stops that never asked for one. Given midnight as the cutoff, the deterministic solver correctly answered that nothing fits before dark, and the model faithfully relayed its own mistake.
The in-loop auditor is blind here — it can only check the constraints the model actually passes — a tool given nonsense answers honestly. The measurement caught it anyway, because the grader knows each scenario's right answer independently. That is the whole thesis: you can measure an agent honestly only when the task has a checkable finish line the model can't talk its way past.
Reading the caveats honestly
Forty runs cannot prove 100%. Every perfect score here carries a 95% interval reaching down to 91%: four models are indistinguishable at this sample size, which is the finding, not a tie to be broken. gpt-oss-120b is accurate but slow — 95%, and a median plan of 128 seconds against 15 for the default, because it emits only ~25 output tokens a second and spends over a thousand of them thinking per run. qwen3-8b is the only model whose accuracy is genuinely behind, and it is also 18× the price of the default. Cheap is not the same as small: the cheapest model here is also among the most reliable.
How the test got here
Harder scenarios, and more patience for slow models
This sweep is the latest of several, and the test moved in two directions at once. It got harder: the scenario library grew to eight, topped by grand-tour — six dogs carrying eight constraint types in one outing, where nearly every remaining failure now lives. And it got more patient, because an impatient harness measures the network instead of the model.
01 Time to think
An earlier sweep cut runs off at 90 seconds and recorded slow-but- working models as timeouts. A run now gets a 300-second budget with a 120-second ceiling per model call, which is why a 128-second median shows up as a cost rather than a failure.
02 Room to answer
A reply may run to 4,000 output tokens. A six-dog plan legitimately needs well over a thousand, and a ceiling below that clips the tool call mid-JSON. Output tokens bill as produced, so headroom is free.
03 Patience with throttling
Rate-limited calls are retried up to six times with exponential backoff, honoring the provider's own Retry-After. A busy provider costs seconds, not a failed run.
04 Quotas kept out of it
Street-distance lookups are cached between runs, so a sweep asks the map service about a dozen questions instead of hundreds — and a run is thrown out entirely if its route ever falls back to straight-line distances.
How it was measured
A grader that can't be charmed
The task is dog-walk route planning: geocode addresses, solve the visiting order over real streets, check weather for each dog's actual walk window, respect terrain, medication, and comfort constraints, and finish with a structured plan. The numbers here are a single 2026-09-18 sweep of 240 live runs. Runs whose failure belongs to the provider rather than the model are reported separately and left out of the pass rate. Try it live →
01 Deterministic grade
A run passes iff it reaches a schema-validated submit_plan whose feasible flag matches the scenario's known answer. No model judges another. All eight scenarios' answers were checked against the solver before the sweep.
02 Real cost & latency
Dollars, not token estimates. Every model round's OpenRouter usage — prompt and completion tokens plus the actual charge — is summed per plan, so cost-per-run is measured, not modeled.
03 Five runs, real intervals
Each of the 48 model×scenario cells runs five times. Pass rates carry a Wilson score interval, so "5/5" and "50/50" are never mistaken for the same confidence.
04 Failures kept whole
Every failing run's full transcript is archived and sorted into a taxonomy from the event stream — veto_livelock, schema_thrash, prose_stall, fabricated_feasible, false_infeasible, backend_error, timeout. The rejects are the content.