Three cheap eval runs beat one flagship pass

When an LLM system underperforms, the default move is to reach for a bigger model. It’s the visible dial. The pricing page makes it look like the quality dial too: pay more per token, get more correct answers. A benchmark I read this week makes a strong case that for a whole class of tasks, that dial is the wrong one to turn first.

The short version: three pooled runs of a cheap model matched the single-pass recall of models costing an order of magnitude more. And the same model, on the same task, swung from finding 7 bugs to finding 13 between identical runs. If your evaluation can’t see that swing, you don’t know what your system does. You know what it did once.

The benchmark

Aikido ran 13 models from OpenAI, Anthropic, xAI, Google, and open-weight vendors against 26 known CVEs from GitHub advisories, inside their production code-analysis harness. Each model was pointed at the vulnerable snippet and scored pass@3 on whether it rediscovered the known flaw.

The leaderboard part went how leaderboards go. GPT-5.6 topped recall at 23 of 26, Kimi K3 tied it, grok-4.5 found 20, and the Claude Opus models landed between 15 and 18. Fine. If that’s all you take away, you’ve read another ranking.

The operationally useful finding is underneath: single runs miss bugs that other runs of the same model catch. Pooling three passes of gpt-5.4-nano reached 18 of 26 for roughly $170 of inference, which ties or beats what the Opus-tier models found in a single pass. And claude-haiku-4-5 found 7 CVEs on one run and 13 on another, same task, same code, nothing changed.

One caveat before building on this: it’s vendor content, run inside Aikido’s own product harness, so the absolute numbers serve their pitch. The variance finding is the part I trust, because someone with no product to sell measured the same thing.

The swing is the finding

Dan Luu ran his own model-comparison experiments and found that one standard deviation between runs of the same condition was around 7.5%, wider than the gap between his best and worst conditions. His conclusion was blunt: look at a small number of individual runs and pretty much any conclusion is possible. For frontier models, you can flip a benchmark ranking by swapping a handful of tasks out of a hundred, because most tasks are trivially all-pass or all-fail and a small subset decides the order.

Put the two together and single-number benchmark scores stop meaning much. “Haiku found 7” and “Haiku found 13” are both true single-run statements about the same model on the same task. A dashboard that reports either number as the score is lying to you by omission.

I keep coming back to this from the support side, because Tier 3 work teaches you the same lesson with different furniture. Nobody who has debugged production systems trusts one run of an intermittent failure. A bug that reproduces once in three attempts is a real bug; a fix that survives one attempt is not a verified fix. You run it again. You run it until the behavior is a distribution you can reason about, because the alternative is shipping on a coin flip that happened to land well.

Model outputs are intermittent failures with a marketing budget, and pass@3 is reproduction runs under a new name.

The harness is the lever

If run-to-run variance inside one model is bigger than the gap between model tiers, then the thing that determines your system’s real-world recall is whatever sits around the model: how many passes you pool, how you filter false positives, and whether you’re pointing the model at the right code in the first place. Aikido’s own conclusion says it plainly: treat the harness, not the raw model tier, as the variable that aims reasoning at the right place.

Luu found the same thing from the other direction. In his tests, a weaker model inside a pipeline with good false-positive filtering beat a stronger model with none. The setup around the model was worth at least as much as the model.

There’s a ceiling, and it’s worth being honest about it. In the Aikido data, every model caught the obvious injection and deserialization flaws. Only the strongest traced an indirect SQL injection through an unescaped ORM column alias. Deep multi-step reasoning is still something you buy with model tier, and no amount of pooling cheap runs conjures it. Flagships earn their price on that class of problem. Most of the gap people try to close with a flagship subscription, though, is plain variance, and variance is cheap to buy down.

What this changes in practice

Three changes this makes to my own eval work.

Score distributions, not runs. If your eval suite runs each case once and reports a number, the number is noise wearing a suit. Run the suite multiple times and look at the spread. A prompt change that moves the mean less than the run-to-run spread didn’t do anything, no matter what the single before/after comparison says. This is the eval equivalent of refusing to close a ticket on one clean reproduction.

Price pass@k against the tier upgrade. Before defaulting a hard task up a model tier, cost out pooling two or three runs of the cheaper tier with a vote or a union over results. In Aikido’s data that trade bought Opus-tier recall for nano prices. Your task will have its own numbers, which is exactly why it belongs in your eval suite as a measured comparison rather than an assumption.

And spend the saved money on the harness itself: false-positive filtering, better retrieval of what the model looks at, verification steps that force a claim to be checked before it’s reported. Every one of those moved the numbers in Luu’s testing more than model choice did. They’re also the parts you own and can iterate on, unlike the model.

The model tier is the headline. The harness is the job. If ten thousand support tickets taught me to distrust an answer that was only checked once, this benchmark says the same discipline is now a line item on your inference bill, and it’s the cheapest one there.

---

Sources: Aikido’s 13-model CVE benchmark (via TLDR InfoSec, July 2026; vendor-run, caveat applied) and Dan Luu’s model-measurement notes.