July 4, 2026 by Toyon
Toyon Finds 3.19x More Bugs Than GPT-5.5
SM-100 Bug Benchmark
3.19×
+57 bugs with Toyon
On July 4, 2026, we ran the full SM-100 benchmark with Toyon using GPT-5.5 in deep source-scan mode.
Toyon found 83 of 100 target bugs. The reference agent baseline found 26 using the same GPT-5.5 model.
83 hits versus 26
| System | Model | Hits | Misses | Errors | Hit rate |
|---|---|---|---|---|---|
| Reference agent baseline | GPT-5.5 | 26 | 74 | 0 | 26% |
| Toyon, deep-scan run | GPT-5.5 | 83 | 17 | 0 | 83% |
Toyon found 57 more bugs than the reference agent, or 3.19 times as many.
How SM-100 works
SM-100 is a source-level bug-finding benchmark built by Bismuth. It collects 100 real bugs from 84 public repositories spanning Python, TypeScript, JavaScript, and Go. Every bug in the set was found and fixed in the wild, and the benchmark is deliberately limited to objective defects such as explicit security issues or logic errors that can cause data loss or crashes. It excludes feature requests, optimizations, and style or design preferences.
Each task points at a real repository state and a target issue. A run scores a hit only when the reported bug location lands within the benchmark's accepted gold-line window. Unlike feature benchmarks, SM-100 tests maintenance: the agent has to search across a codebase for a bug without knowing where it lives. Agents that score well on SWE-bench routinely score far lower here.
For more on the benchmark's design and the original cross-agent results, see Bismuth's blog and this independent write-up of SM-100.