Yesterday I ran my drafting cycle over 16 of my own topics and read the reports. The draft of a blog essay about keeping state in files came back with a critic score of 2 out of 10. In this setup that means it does not smell of a model and goes to me for reading. The mechanical checks on the same file counted 10 hits, 8 of them a comma before and.
That comma is not something the critic has to work out from my style. It is hard check number 3 in the critic’s own prompt, written there as Comma before and / or → 0. And the critic did see one of the 8: its remarks on that draft quote a comma before and and call it a violation of the hard check. It named the thing and still returned a 2. The sentence I use to explain this cycle says anything with a definite answer gets decided in code before the model is asked anything. The model keeps only the judgement calls. That line sits in the header comment of tells.ts and I repeat it whenever somebody asks how the drafting works. I had never measured it.
The cycle and the setup
A draft goes through src/lib/tells.ts first. There are 19 checks there. About half landed after a correction I made by hand: no dash in English, no comma before and, digits for counts. The rest arrived as a rubric I never tested. Then the draft goes to the critic in prompts/_critic.md, which returns remarks and a score from 0 to 10 where lower is better. At 3 or under it stops and waits for me. Above 3 it gets rewritten, up to 2 rounds.
Everything below ran with claude-opus-5 through the claude CLI 2.1.235 on node 22.23.1, against tells.ts at commit 0f0b511. The runs happen inside this repository on purpose. A clean directory is the right call when the model is what you measure. In August I counted 0 dashes across 30 machine texts and reported it as a property of the model. It was this repository’s own ban being obeyed. The same model from an empty folder gave 6.10 dashes per 1000 words. Here I measure the cycle with the repository’s own CLAUDE.md in the context, because that is how it works every day.
16 drafts through the real pipeline
16 topics, 12 approved for X and 4 blog essays. All 16 fired the checks: 125 hits in total, between 1 and 20 per draft. The critic gave scores from 2 to 5 with a mean of 2.9, so 13 of the 16 sit at or under the threshold. The loop would stop there and hand me the file.
That total needed a repair before it meant anything. One draft came back wrapped end to end in a single code fence. The check for welded hyphens strips fenced blocks before it looks, so that draft counted as 0 words of prose. Peeling off the one outer fence is the difference between 123 hits and 125.
62 of the 125 hits are that one comma. The correlation between the critic’s score and the hit count looks encouraging at r = 0.51 over 16 points. Per 1000 words it is r = -0.21. Hits and draft length go together at r = 0.77. The score against length is r = 0.41, so that first number is mostly telling me that long drafts are long.
Feeding it violations on purpose
So I held the text still and varied one thing. A clean piece of 857 words that fires nothing, with k violations of a single rule injected into it, k doubling from 1 up to 8. Every version went to the critic 3 times, the clean base included.
Signposting behaves the way I hoped. At 8 injections it reaches 6.0, well above the threshold, named in every run. Dashes reach 4.0. My comma rule finishes at 2.7 with 8 of them in the text. That is inside the passing band, against 2.3 for the base with nothing injected. At 8 commas all 3 runs do name the rule in their remarks while the number stays where a draft goes out to me.
Counting how often the critic names a rule is where I tripped. My first pass matched the dash character inside its remarks, which gave a flattering answer until I looked at what it was matching. The critic writes dashes itself, in 42 of its 90 remarks, in a repository whose first English rule is that there are none. Matching the word dash dropped the answer to 1 of the 4 drafts that had one.
The same text, 12 times
A mean of 2.9 is worth something only if the number repeats. I ran the critic 12 times on one text, then 12 times on the same text with 2 dashes in it.
The clean text scatters over 1 to 3 with a mean of 2.2. The dosed one returns 3 on all 12 runs. The critic wobbles on clean prose and locks on dirty prose. That is backwards from what I assumed, but 1 text and 1 rule is thin evidence.
Which report a rewrite should get
Here the question turns into a decision. A forced rewrite round on 15 drafts, 2 arms, the same model in both. Arm A gets the critic’s remarks. Arm B gets the machine report, meaning rule names and counts. Both arms start from the text as the cycle left it, so their opening counts are lower than in the table above.
Arm B answered on 8 of the 15 drafts and took 4 of those 8 down to 0 hits, introducing no new rule anywhere. Arm A answered on 9 and took none of them to 0. In 4 of the 9 it introduced a rule that was not in the draft before. One essay went from 4 hits to 8 while doing exactly what the critic asked for. The missing cells are the CLI answering HTTP 429 on a spend limit. A tenth A cell finished and was lost, because the script writes its file after both arms and the process died in between.
Where the checks fail, on a list I typed myself
The check named spelled-number fires on counts written as words. The rule is a sentence in a voice guide. The check that enforces it is a list of 15 words.
thirteen through nineteen are missing from that list, so are sixty, seventy, eighty and ninety. Across the 22 essays that were on this site before this one, 22578 words of prose, spelled-number fires zero times. 8 counts written as words are sitting in text I published. 6 of the 8 are in one essay about a multiply that never runs, where I wrote sixteen in words 5 times and fifteen once.
What I changed and what I did not check
The rewrite step now gets the machine report alongside the critic’s remarks, on the strength of arm B. Both at once is a third case nobody measured. I took it because the round only starts when the critic scored above 3, so throwing its remarks away would leave the reason for the round unanswered. The README line survives in a narrower form. The checks answer what somebody already typed into a list. The critic answers the rest. It can be handed a hard check in its own prompt and still pass a draft that breaks it 8 times. Neither one is a gate for the other, which is the part I had wrong.
I did not check whether the critic’s number has anything to do with a text being good to read. There are no human ratings anywhere in this. 3 runs per cell is thin. This is a single prompt on a single model.
The counting has a hole I have not closed either. The drafts here were judged as whole files while the critic judges the post inside them. On the 10 X drafts where the post sits in a fenced block, the post halves fire 25 times against 47 in the notes. Those notes go to nobody. The 2 halves do not add up to the count over the whole file, because one of the checks strips fenced blocks before it looks.