In August I gave Anthropic models 10 pieces of SaaS arithmetic with every tool taken away. One of them was a cash forecast: $420,000 in the bank and $40,000 spent every month, with revenue starting at $18,000 and growing 11% a month. Opus 5 got it to the dollar. Haiku 4.5 did not. I quoted its miss in public twice. On 12 August on X it was $5,775 short. On 18 August on LinkedIn, after a rerun with 4 models, it was $8,451 off. I added that the number still looks right. That was my point: nothing in the answer says it is wrong.
That example was convenient and I used it as a fact about Haiku. But I had 2 samples, one per run. They did not even agree on the sign. I never asked what a third one would look like.
There is also a smaller embarrassment. Both posts call it a forecast over 5 years, while the question says 36 months. The results.json of the run says months: 36. The table in the public repository says 60. 2 other rows of that table have wrong month counts too.
150 calls on one question
On 12 October I ran the same question many times. Claude Code 2.1.292, claude -p from an empty folder, --tools "" plus an empty MCP config, so the model has nothing to compute with. The answering model is checked in modelUsage, not taken from the flag. The truth is computed in Python Decimal: the answer is $5,822,959.30.
First the original prompt, one line ANSWER: <number>, 30 times for Haiku and 10 for Opus. Opus was exact 9 of 10, the tenth was $4.19 off. Haiku was exact 0 of 30. 23 of its answers were within $17k of the truth, median miss $2,556, 9 above and 14 below. The other 7 sat around $6,507,000, which is $684k too much.
I first thought the 7 big ones were a misread question and started counting months. 37 months of revenue gives $6,593,685, 35 months of spend gives $5,862,959. Neither fits. What fits is dividing the sum of the series by 0.10 instead of 0.11:
420000 + 18000 * (1.11^36 - 1) / 0.10 - 40000 * 36 = 6,507,255All 7 answers are within $6,600 of that number. This prompt has no working in the reply, so for these 7 it is a match of numbers and not something I saw.
Where 0.10 comes from
To see it I asked for the working, 20 more calls. 11 of 20 ended near $6.5M, more often than with the short prompt. In every one of the 11 the working divides by 0.10 or 0.1. All 9 good ones divide by 0.11. In 6 of the 11 the line just before it has the denominator written out correctly:
Total Revenue = $18,000 × (1.11³⁶ - 1)/(1.11 - 1)
= $18,000 × (1.11³⁶ - 1)/0.10The model writes the general formula, puts in 1.11 and on the next line 1.11 - 1 is 0.10. In 3 of the 11 the step is not written at all and the working goes straight to 0.10. It is a slip of one digit in the easiest step of the problem. That one digit is what multiplies the revenue by 1.1.
The $8k was the other error
The August misses, $5,775 short and $8,451 over, are not this slip. They look like the 23 near answers. The whole forecast depends on one number, 1.11 to the power 36, which is 42.818. The revenue multiplies it by 18000 / 0.11, so an error of 0.001 in the power costs $164 in the answer.
I checked that with a prompt that asks for 3 lines: the power, the revenue, the answer. 30 calls. The powers went from 42.736 to 42.929. Median miss of the answer was $742 and the error of the model’s own power times 163,636 explains $741 of it. In 27 of 30 the answer matched what its own power gives within $10, in the other 3 it was off by $200 to $1,636. I cannot tell from this whether the model computed the power first or adjusted the printed power to an answer it already had. What I can say is where the printed power goes wrong: the first wrong digit is in the third decimal in 14 of 30, in the first or second in 13. Asked for the bare power alone 20 times, Haiku never gave 42.818084. In forecast dollars the spread was −$32,741 to +$41,223.
In this prompt the 0.10 slip did not happen at all, 0 of 30. I do not know why. One guess is that writing the revenue as its own line makes the model compute the division separately. I did not test that and 30 calls is small.
Asking it to say when it is unsure
The last thing I wanted to know is whether Haiku can tell. Same short prompt plus one sentence: if you are not certain the number is exact to the dollar, add a second line UNSURE. 30 calls.
Haiku said UNSURE 7 times. All 7 were on answers that were off by $2,817 or less. 4 answers came with the +$684k slip. None of them was marked. Opus, given the same sentence, marked 3 of 10, while all 10 of its answers were exact.
So the flag exists, but it falls on answers that are close anyway. Many close answers have no flag either. The 4 worst answers came out with no flag at all.
What changes for me
The example I used in August was real but small. On this question a Haiku answer is usually a few thousand dollars off, sometimes up to $17k. In the prompt with 3 lines the printed power explains that drift. In 7 of 30 calls on the original prompt it was $684k off because 1.11 - 1 became 0.10. I quoted the drift and did not see the slip, because I looked at 1 answer per run.
For anything I would act on, the model should not do this division at all. I said that in August as well. Now I know which step fails and what it looks like on the page: (1.11 - 1) on one line and 0.10 on the next.
What I did not check
Only Haiku 4.5 and Opus 5, only this one question. Other numbers may not have a 0.10 sitting next to 0.11. Every arm is one session of 10 to 30 calls and I did not repeat any of them, so 7 of 30 against 0 of 30 against 11 of 20 is a hint at most. For the big answers I tried 2 other explanations, both about month counts, before 0.10 matched. Tools were off on purpose, while anyone would really run this with tools.