July 4, 2026
Toyon on SM-100: Over 3x Better Than GPT-5.5
A July 2026 SM-100 run shows Toyon finding 83 of 100 benchmark bugs with GPT-5.5, compared with 26 of 100 for a reference agent baseline using the same model.
As of July 4, 2026, we ran the full SM-100 benchmark with Toyon using GPT-5.5 in deep source-scan mode.
The result: Toyon found 83 of 100 target bugs. A reference agent baseline, run with the same GPT-5.5 model, found 26 of 100. The reference agent is the baseline agent that ships with the SM-100 benchmark, so it serves as the benchmark's own point of comparison.
This is a harness comparison, not a model comparison. Both runs used GPT-5.5 at high reasoning effort; the difference is the system wrapped around the model.
Headline Results
| System | Model | Hits | Misses | Errors | Hit rate |
|---|---|---|---|---|---|
| Reference agent baseline | GPT-5.5 | 26 | 74 | 0 | 26% |
| Toyon, deep-scan run | GPT-5.5 | 83 | 17 | 0 | 83% |
The Toyon run is 57 hits ahead of the reference agent baseline. Expressed as a multiple, Toyon found 3.19x as many benchmark bugs as the reference baseline in this comparison.
About SM-100
SM-100 is a source-level bug-finding benchmark built by Bismuth. It collects 100 real bugs from 84 public repositories spanning Python, TypeScript, JavaScript, and Go. Every bug in the set was found and fixed in the wild, and the benchmark is deliberately limited to objective defects such as explicit security issues or logic errors that can cause data loss or crashes. It excludes feature requests, optimizations, and style or design preferences.
Each task points at a real repository state and a target issue, and a run is scored as a hit only when the reported bug location lands within the benchmark's accepted gold-line window. The benchmark is hard because it measures maintenance rather than feature work. An agent has to find a latent bug with no prior knowledge of where it lives, which demands cross-file reasoning and a systematic search of the codebase. Agents that score well on feature benchmarks like SWE-bench routinely score far lower here.
For more on the benchmark's design and the original cross-agent results, see Bismuth's blog and this independent write-up of SM-100.
What Changed
The reference baseline used GPT-5.5 through the reference agent that ships with SM-100, a straightforward harness the benchmark provides as its standard point of comparison. Toyon used GPT-5.5 through its source-aware product QA harness with deep scanning enabled. Both runs used the same model at the same high reasoning effort, so the only variable is the harness. For this run, the SM-100 runner forced the Toyon model and scan mode so all tasks used:
| Setting | Value |
|---|---|
| Model | openai/gpt-5.5 |
| Reasoning effort | High |
| Scan mode | deep |
| Tasks | 100 |
The Toyon run produced no infrastructure errors and no timeouts. Every task completed as either a hit or a miss.
What the Misses Look Like
The 17 misses were not tool crashes. The logs for the miss set were clean: no authentication failures, no context failures, no timeouts, and no Toyon runtime errors. They were ordinary benchmark misses: Toyon reported either the wrong bug, the right subsystem but wrong target area, or a nearby issue outside the scorer's accepted line window.
| Miss category | Count |
|---|---|
| Near miss: right bug family but outside scorer window | 2 |
| Right file, wrong bug | 8 |
| Right subsystem, missed target area | 6 |
| Did not touch gold file | 1 |
| Runtime or tool failure | 0 |
This matters because it means the remaining gap is mostly quality of investigation and precision of final reporting, not benchmark execution reliability.
Why Toyon Helps
The result suggests that GPT-5.5 benefits materially from a harness that makes source investigation more systematic. Toyon gives the model a deeper operating loop around repository orientation, source-aware decomposition, child-agent investigation, artifact capture, and structured issue reporting. On SM-100, that scaffolding moved the same underlying model from 26 hits to 83 hits.
Put differently: the model already has a lot of latent debugging ability. The harness determines how reliably that ability is converted into a scored bug report.
Bottom Line
On this SM-100 comparison, GPT-5.5 found 26 of 100 bugs through the reference agent harness and 83 of 100 bugs through Toyon. The model is the same. Toyon is the difference.