Impact
Measured, not claimed
8 bugs were planted in the ShopLite demo repo; IBM Bob worked 8 of them from the customer's report to a failing test, a culprit commit and a verified fix. Every number is computed from the case events and Bob's task log — docs/benchmark.md.
Culprit commit matches the answer key
8/8
- 2 found by git bisect live, during the run
- 5 found by git bisect in a follow-up Bob task
- 1 named by the Historian from git history
Bugs proven
8/8
0 unproven · RED test → fix → all tests green
Median time to a failing test
4m 02s
from the customer's report to a RED reproduction
Fully live run (bug #5), start to proof
7m 44s
culprit by bisect at 4m 01s · 0 human prompts
Bobcoins per fix
2.93
median · range 2.39–5.36
Compared with one-shot models
This compares evidence, not model quality. Bob worked inside the repo, could run its tests and produced a proof anyone can re-run; 7 models (five on IBM watsonx.ai, plus Gemini 3.1 Pro and Claude Opus 4.6) got the source in the prompt and answered each bug once, without running anything. Every answer looked finished, with a fix and a test, yet 18 of 56 did not fix the bug, and only running them showed which. Cimex Fix is that run, automated for every fix. The best model, gpt-oss-120b, got 8/8 fixes with proof, matching Cimex Fix's 8/8, and that is only known because the checks ran. Gemini's and Opus's one miss (#1) changed the behaviour instead of restoring it: an empty coupon field now shows an error instead of meaning “no coupon”. Their own tests passed because they asserted their own fix design, which the Reproducer's symptom-only rule forbids. One script scores every answer, Bob's included, and the answer key's independent probe decides whether a bug is fixed.
All 56 one-shot answers
- named a culprit and returned a fix and a test
- 56/56
- fixed the bug (independent probe)
- 38/56
- came with a test that proves the fix
- 30/56
- broke existing tests
- 5/56
| Contender | Fixed (independent probe) | Nothing broken | Own test RED→GREEN | Fix with proof | Culprit commit | Cost and time per bug |
|---|---|---|---|---|---|---|
| Cimex Fix (IBM Bob)Bob pipeline: runs tests and git bisect | 8/8 | 8/8 | 8/8 | 8/8 | 8/84/8 during the run | 2.93 Bobcoins median per bug 13m 45s median to proof |
| Granite 4 H Smallone answer, cannot run code | 2/8 | 6/8 | 1/8 | 1/8 | 0/8 | 7,296 tokens 8.3 s per answer (median) |
| Llama 3.3 70Bone answer, cannot run code | 4/8 | 8/8 | 2/8 | 2/8 | 6/8 | 7,204 tokens 13.05 s per answer (median) |
| Llama 4 Maverickone answer, cannot run code | 5/8 | 6/8 | 3/8 | 3/8 | 4/8 | 7,461 tokens 9 s per answer (median) |
| Mistral Small 3.1one answer, cannot run code | 5/8 | 7/8 | 3/8 | 2/8 | 5/8 | 7,907 tokens 9.9 s per answer (median) |
| gpt-oss-120bone answer, cannot run code | 8/8 | 8/8 | 8/8 | 8/8 | 6/8 | 8,474 tokens 14 s per answer (median) |
| Gemini 3.1 Pro (High)one answer, cannot run code | 7/8 | 8/8 | 8/8 | 7/8 | 7/8 | 46,871 tokens 75.45 s per answer (median) |
| Claude Opus 4.6 (Thinking)one answer, cannot run code | 7/8 | 8/8 | 8/8 | 7/8 | 7/8 | 41,157 tokens 41.2 s per answer (median) |
| Bug | Cimex Fix | Granite 4 H Small | Llama 3.3 70B | Llama 4 Maverick | Mistral Small 3.1 | gpt-oss-120b | Gemini 3.1 Pro (High) | Claude Opus 4.6 (Thinking) |
|---|---|---|---|---|---|---|---|---|
| #1 Empty coupon field → total shows ₹NaN | fix with proofyesculprityes | fix with proofnoculpritno | fix with proofnoculpritno | fix with proofnoculpritno | fix with proofnoculpritno | fix with proofyesculpritno | fix with proofnoculpritno | fix with proofnoculpritno |
| #2 Totals off by ₹0.01 | fix with proofyesculprityes | fix with proofnoculpritno | fix with proofnoculprityes | fix with proofnoculpritno | fix with proofnoculpritno | fix with proofyesculpritno | fix with proofyesculprityes | fix with proofyesculprityes |
| #3 Delivery date one day early after 8 PM IST | fix with proofyesculprityes | fix with proofnoculpritno | fix with proofyesculprityes | fix with proofnoculprityes | fix with proofyesculprityes | fix with proofyesculprityes | fix with proofyesculprityes | fix with proofyesculprityes |
| #4 Pagination drops the last product | fix with proofyesculprityes | fix with proofnoculpritno | fix with proofnoculprityes | fix with proofyesculprityes | fix with proofnoculprityes | fix with proofyesculprityes | fix with proofyesculprityes | fix with proofyesculprityes |
| #5 Search became case-sensitive | fix with proofyesculprityes | fix with proofyesculpritno | fix with proofyesculprityes | fix with proofnoculprityes | fix with proofyesculprityes | fix with proofyesculprityes | fix with proofyesculprityes | fix with proofyesculprityes |
| #6 Negative quantity accepted → negative total | fix with proofyesculprityes | fix with proofnoculpritno | fix with proofnoculpritno | fix with proofyesculprityes | fix with proofnoculprityes | fix with proofyesculprityes | fix with proofyesculprityes | fix with proofyesculprityes |
| #7 Double-click "Place order" → two orders | fix with proofyesculprityes | fix with proofnoculpritno | fix with proofnoculprityes | fix with proofyesculpritno | fix with proofnoculpritno | fix with proofyesculprityes | fix with proofyesculprityes | fix with proofyesculprityes |
| #8 Coupon applies twice → discount over 100% | fix with proofyesculprityes | fix with proofnoculpritno | fix with proofnoculprityes | fix with proofnoculpritno | fix with proofnoculprityes | fix with proofyesculprityes | fix with proofyesculprityes | fix with proofyesculprityes |
- The models got every file under src/ and the git history up front, but not the tests and no way to run code; Bob started from the report, explored the repo itself and could run its tests. Bob read the screenshots as images; the models got their visible text.
- One answer per model, asked on 2026-09-26: a single sample, not an average. watsonx.ai models ran at temperature 0 (us-south); Gemini and Claude Opus ran through Google's Antigravity CLI (headless, one fresh session per bug, default temperature, no commands run). Prompts and raw answers are in the repo.
- Bob's culprit count is the final result, including follow-up bisects; 4/8 were named during the runs.
- In Cimex Fix, Granite writes the plain-English summaries and powers /triage; it does not write the fixes.
- Control: every probe fails on the unfixed code.
Time to proof, per bug
In run order. Runs #3 and #8 include time spent waiting on a human while a pipeline bug was being fixed; bug #5 ran after the fixes. Hover or focus a bar for details.
- #4 Pagination drops the last product
- #6 Negative quantity accepted → negative total
- #1 Empty coupon field → total shows ₹NaN
- #3 Delivery date one day early after 8 PM IST
- #8 Coupon applies twice → discount over 100%
- #5 Search became case-sensitive
- #7 Double-click "Place order" → two orders
- #2 Totals off by ₹0.01
Every run
| Bug | Intake | RED test | Proven | Human prompts | Bobcoins | Culprit commit | Answer key | How it was found |
|---|---|---|---|---|---|---|---|---|
| #4 Pagination drops the last product | issue text | 6m 09s | 14m 39s | 0 | 2.84 | 882203d refactor(catalog): compute pagination bounds explicitly | Match | bisect (follow-up Bob task) |
| #6 Negative quantity accepted → negative total | server log | 2m 58s | 7m 40s | 0 | 2.78 | c3fb68f refactor(cart): accept numeric strings for quantities | Match | Historian (git history) |
| #1 Empty coupon field → total shows ₹NaN | screenshot | 6m 19s | 21m 06s | 0 | 4.91 | c3f394a refactor(coupons): extract coupon rule parser | Match | bisect (follow-up Bob task) |
| #3 Delivery date one day early after 8 PM IST | PDF (QA report) | 5m 06s | 43m 49s | 3 | 2.39 | b4369d9 perf(delivery): compute IST date without Intl.DateTimeFormat | Match | bisect (follow-up Bob task) |
| #8 Coupon applies twice → discount over 100% | screenshot | 4m 32s | 34m 01s | 4 | 5.36 | 185f248 feat(cart): allow stacking multiple coupon codes | Match | bisect (follow-up Bob task) |
| #5 Search became case-sensitive | issue text | 3m 12s | 7m 44s | 0 | 3.06 | 649b24a perf(catalog): memoize search index | Match | bisect (live, during the run) |
| #7 Double-click "Place order" → two orders | issue text + log | 2m 25s | 8m 02s | 0 | 2.54 | 985e247 refactor(orders): derive duplicate-order check from order history | Match | bisect (follow-up Bob task) |
| #2 Totals off by ₹0.01 | issue text | 3m 32s | 12m 51s | 0 | 3.03 | 98e2fce feat(cart): sum multiple line items in cart totals | Match | bisect (live, during the run) |
How to read these numbers
- Times run from the moment the Lead opens the case; “proven” is the last publish of the Proof of Fix.
- Early runs hit a bisect bug: Bob passed the repo path with a lowercase drive letter, so Vitest loaded twice and every commit was skipped. After the fix, a follow-up bisect confirmed those culprits; bug #5 found its culprit live.
- Human prompts are messages typed after the first one. Bobcoins come from IBM Bob's task log; building the Bob pack cost 4.83 more, and all Bob work so far totals 32.64.
- There is no manual baseline, so no human-versus-Bob time comparison is claimed.
Generated 2026-09-27 07:35 UTC.