Code review here runs in 3 stages. N agents read the file and report defects, K skeptics try to refute every report, one agent merges what survives into a list. I set it up that way because the claude CLI I run agents with ships a research workflow built on that shape. Its phases sit in the binary as plain strings: fan out the searches, extract falsifiable claims, 3 votes of adversarial verification per claim with 2 refutes to kill, then merge the semantic duplicates. I copied that into code review without asking whether code review wants it. I have been paying for all 3 stages since and I never once picked N or K on evidence.
This week I measured it. 191 model calls and $11.76, counting a control round I threw away. The stage I would have defended hardest took $4.57 of that and never once disagreed with itself. What I did not expect is how often the thing that turned out to be wrong was my own apparatus rather than the agents. It happened 5 times. The worst of the 5 nearly went out as the headline of this post.
The file
clean.mjs is a metrics collector of the kind this workspace actually needs. 184 lines: a semaphore over a FIFO queue, retries with backoff and jitter, a cache key, a nearest rank percentile and a daily summary line. A second script plants 12 defects into it by literal string replacement and the file the agents read comes out at 176 lines. A ledger records where each defect went. There is one file per defect carrying that defect alone.
Every defect is then proven by running the clean file against the file that carries only it. Proving them against the mutated file is the obvious thing to do and it gives the wrong answer. One defect can hide behind another there. One of mine does.
Every stage runs from a fresh temporary directory. No CLAUDE.md on the path and no tools. I checked that by hand rather than had the harness report it. The MCP config is empty. The operating manual of this repository is 889 lines of rules I wrote for myself. It goes into the prompt silently if the working directory is the repository. Node is 22.23.1, the claude CLI is 2.1.235, the model is claude-opus-5 on 4 cores.
Finding
8 finders got one identical prompt, 4 running at a time. They came back with 108 findings for $0.76. My scorer matches a finding to a planted defect when the reported line falls inside the planted span widened by 2 on each side. It said the mean finder catches 10.25 of 12 and the one nobody found is the semaphore.
I read the 108 claims because that last part looked wrong. All 8 finders had found the semaphore. 6 of them reported it at line 25 and 2 at line 26. That is the one live++ the file still has. I had deleted the other one at line 18 and I had scored them on the position of my own edit. The window also handed 2 finders credit for defect 12. Their claims on those lines are about other things: a 200 response with a body that is not JSON and a journal directory nobody creates.
So nobody found defect 12 and in this file no failing status reaches it. Defect 12 removes the status check after the retry loop. Another planted defect removes the check that stops a bad status from being retried at all. The loop now either returns a good response or throws. I swept 9 failing statuses through the file and 0 of them reach the response body. Through the file carrying only defect 12, 6 do.
Corrected by reading, all 8 finders found the same 11 defects. On the planted defects, in this round, 7 of the 8 bought me nothing. Both ways of scoring end at 11 of 12 with different defects inside them.
That is one round, so I ran the 8 finders again on the same file with the same prompt. 108 findings, the same total, which I take as a coincidence. Of the 26 lines named in either round only 18 were named in both. The union was 11 again and the same 11, but one finder came back with 10. It missed the retry policy that retries a 400 and it really did miss it. There is nothing about retry policy anywhere in its 14 findings. The prompt allows 15, so nothing was crowded out.
The file with nothing planted
The same 8 finders on the original clean.mjs returned 0 findings. 8 empty lists out of 8. I wrote that down as the false positive rate and moved on, pleased with the code.
clean.mjs opens with a docblock whose lines 4 to 6 say that this is the clean version. They also say that another script plants the defects into it. Line 4 is a blank comment line. The other 2 carry the statement. The mutation script strips exactly those lines when it writes the file the finders read. So in my control I had told 8 reviewers, in the first thing they laid eyes on, that the defects live in some other file.
I cut the lines, ran the same 8 finders and got 74 findings. Not one empty list. That looked like the answer, it was nearly the headline of this post and it was still not clean. Cutting 3 lines moves every line below them up by 3. The prompt hands the model a listing with the numbers on it. The finders had seen 2 changes where I meant them to see 1.
So I wrote a third file. The 2 lines that carry the statement, replaced in place by 2 that describe what the code does. Counters are pulled once per post and cached in the journal. The daily summary reads that back. 184 lines like clean.mjs, every line number where it was, every byte of code untouched. diff gives back those 2 lines and nothing else. 62 findings from the same 8 finders. 1 of the 8 still came back empty. There was an earlier run of that same file which I threw away because 2 finders hit a 429. The 6 that answered returned 59 findings between them and not one empty list.
Whether those findings are any good is a question I can only settle by running the file. I collapsed the 74 into 16 distinct claims and decided each one with a script. All 16 hold and they cover all 74. 1 of the 16 is only half decided by a run and I come back to it at the end. Here are 5 of them. A cap of 0 deadlocks the semaphore. The abort timer is cleared before the body is read, so a timeout of 40 milliseconds let a body read of 120 milliseconds finish. cacheKey("a", "b c", ["views"]) and cacheKey("a b", "c", ["views"]) both return 0f7d1e1c5f119f46. One failing post makes Promise.all discard every row already collected. And summaryLine builds its date prefix out of a filename, so it hands a person the string 2026-09-05.md: 12 posts, 400 views, p90 90. My clean file was not clean.
The line window says 60 of the 62 findings from the third file sit within 2 lines of something those 16 claims already decided. I did not believe it this time. Reading them, 4 findings are about requests in flight that nobody deduplicates. The window files those under the claim that a journal hit of any age comes back without a request. What decides them is a different claim, that 2 posts on the same key both miss the cache and both fetch. Another 4 are about fields never reaching the request. Those 4 the window scatters over 2 other claims, when the one that settles them says fields changes the cache key and nothing else. The 16 claims do decide all 8 of these findings. The window is wrong about which claim decides which.
What I keep coming back to is readJournal. 5 lines, an unguarded JSON.parse, byte identical in all 4 files. In the mutated file 7 of 8 finders flagged it and 7 of 8 again in the repeat round. None of the 8 flagged it in clean.mjs, where it sits on line 83. 7 of 8 flagged it in the third file, where it also sits on line 83. I had an explanation ready for the first pair of numbers, something about the rest of the file changing what looks suspicious. It survives that pair, because the mutated file really does carry 12 defects. It does nothing for the last 2 of those. Same 184 lines, same line 83, 2 comment lines apart. What moved the answer was a sentence I had written about my own code.
Verifying
For the second stage I needed claims that are false, because almost everything the finders said was true. I wrote 12 in the same register as theirs and falsified each one by running the code. The queue is LIFO, the percentile sorts in place and 10 more. Against them stand the 11 true claims in the finders’ own words, unedited. Then 3 independent verifiers per claim, twice over. The adversarial prompt says to refute and to fall back on refuted when unsure. The neutral one only asks whether the claim is true.
That came to 138 calls and $4.57.
All 3 verifiers agreed on all 23 claims in both modes and the 2 prompts produced identical verdicts on every claim. The vote bought nothing. Neither did the adversarial framing.
That result is narrow because of the way I built the test. My 12 false claims can be refuted by reading 3 to 5 lines of the file. A claim that needs 2 functions held in mind at once is where a second opinion should pay and I have no measurement of that. The 11 true claims are built one per planted defect, so the 18 findings that landed on no planted line never reached a verifier at all. Those are the ones most likely to have been false. That almost everything the finders said was true is an impression I got from reading them, not a number. The number I have is for the control file.
Merging
108 findings across 23 distinct lines, 6 of which one finder saw alone. The merge agent gets the reports. It never sees the file. 3 runs gave 16 entries each at $0.227 a run. Once I had told the scorer about the 2 merges below, they dropped nothing and invented nothing. The prompt tells the merge agent not to drop a defect because only one reviewer saw it, so that property was asked for rather than discovered.
It also merged 2 findings that sit 76 lines apart: the journal write that is not atomic and the missing directory. It named both reviewers on the merged entry. My scorer called that 2 wrong attributions. A second merge like it, it called a dropped finding. Both stood until I read the entries and told it about them. The scorer only compares line numbers. One merge said so outright in the entry, the other only said both paths.
That is the fifth one. The line window was wrong about 2 defects in opposite directions. It was wrong again about which claim covers which finding in the control. The docblock in my control was wrong about the whole control. My fix for that docblock moved 178 line numbers I had not meant to move. The merge scorer was wrong about both merges. All 5 were mistakes in my apparatus. Every time my apparatus and an agent disagreed, the apparatus was the one that was wrong.
What I did not check
One file, defects planted by replacing between 1 and 6 lines each. Real review work is a diff across files and the defect lives in the interaction. The finding stage ran twice and the merge 3 times. Verify ran once in each of its 2 prompt modes. Each version of the docblock got a single run, so 62 against 0 is one pair of rounds and not a distribution. False claims for the verifiers are mine and they are easy. Of the 16 control claims, 1 is only half decided by a run. I showed that the journal write does not create its directory. The missing rename I took on reading. The rename is what all 8 findings on that line were about. In the third control round 1 finder of the 8 returned an empty list. Why that one, I did not establish. The 2 lines I put in place of the giveaway are not a blank. They describe the journal and the cache, which is where a good part of the 62 findings live. So what I compared is an author saying the file is clean against an author describing the caching. That is not a tell against nothing. The 2 lines I took out also carried 2 signals at once, an all clear and a pointer at another file. I removed them together. Which of the 2 did the work, I did not separate. The $0.094 per finder call that $2.07 rests on comes from the mutated file. The clean file drew longer answers and ran up to $0.29 a call, so $2.07 is the cheap end of what I saw.
Of the 191 calls, 138 are verifiers and 48 are finders over 6 rounds. 3 went to the merge and the last 2 are retries after a 429. $4.57 against $0.76 looks like 6 times the price for the stage that changed nothing. Most of that gap is the second prompt mode and the 12 claims I wrote myself. A review does neither. Priced like for like, one review of this file is $2.07. Finding is $0.76 of that, the merge $0.23 and verifying the 11 claims that survive deduplication $1.09. Those 3 add to a cent more than the total, because each of them is rounded. At 2 finders and 1 verifier the same review is $0.78. I am moving to that, but I am keeping the second finder because of the run that came back with 10.
152 of the 190 JavaScript and TypeScript files in this repository open with a docblock saying what the file is for. I wrote most of them by hand. What I want to measure next is a defect that spans 2 files.