When someone asks me where a language model belongs next to money, I answer with one sentence. The totals come from a deterministic engine and the model only talks about them. On 8 October I measured the forecast side of that split. The invoice side I never measured at all, I just repeated it, so this time I took an invoice.
The setup
I live in Warsaw, so the invoice is a Polish VAT invoice. Every line has a quantity, a net unit price in złoty, a discount and a VAT rate of 23, 8, 5 or 0 percent. The rule is written in the prompt. Each line net is rounded to the grosz half up. Then the lines are grouped by rate, VAT is taken from the sum of each group and rounded again. Gross is net plus VAT.
The engine is about 20 lines of Python 3.12.3 with decimal and ROUND_HALF_UP. 10,000 invoices of 40 lines take from 0.83 to 1.18 seconds, about 0.1 ms per invoice. 3 repeats give the same hash of all results.
The model is claude-opus-5 through Claude Code 2.1.292, started from an empty folder with no tools and no MCP servers. With a fixed seed I generated 10 invoices of 5 lines, 10 of 15 lines and 10 of 40. Every invoice went to the model 2 times. The answer had to be one line of JSON with the 3 totals. A match means net, VAT and gross all equal the engine to the grosz.
Before counting I wrote down what I expected. At 5 lines the model is exact in at least 9 of 10. At 40 lines it is exact in fewer than 7 of 10 and the 2 runs of the same invoice disagree at least once.
The idea that code counts better than the model is old and not mine. PAL and Program of Thoughts in 2022 moved the arithmetic out of the model into a program. I had just never seen it on an invoice.
Small invoices
At 5 lines the model was exact 9 times in each run. At 15 lines it was 9 and 10. The misses were small. One invoice was off by 5 grosz and another by 85 grosz. The third was off by exactly 1.00 złoty net, which moved VAT by 8 grosz.
What bothers me more than the size is that every wrong answer was consistent with itself. Net plus VAT equals gross in all of them, to the grosz. The first check I would write adds up the 3 totals. It passes every one of these.
40 lines, the answer I did not ask for
At 40 lines I got 2 JSON answers of 20. Both were exact. The other 18 were a Python program with Decimal. Some came as a bash code block and some as a call to a Bash tool the model did not have. I had not planned for that at all. So I stopped and wrote down a new rule before counting further: take each program out of the answer and run it.
There were 19 such programs in the whole run, 18 here and 1 more from another part of the experiment. I ran each with python3 -I in an empty temporary folder. All 19 printed the right totals to the grosz.
Then I read further down the answers. In 10 of the 19 the model did not stop at the program and wrote the tool output itself. 9 times it was numbers, 3 of them under a heading Tool Result. Once it was just <redacted>. None of the 9 matched the right totals or even what its own program prints. On one invoice the program prints net 739,434.56 and the invented output under it says 58,923.85. Another invoice got 2 different invented outputs in the 2 runs, 995,552.96 and 1,304,665.13, while the right net is 962,434.28.
For a while I believed 9 of these answers had no code at all. My first extractor found programs in only 10 of the 19 and marked the rest as “no code”. The model had wrapped the same program in 3 different formats and I had not expected more than one.
Told to count by hand
So I sent the same 10 invoices of 40 lines 2 more times with one more sentence in the prompt: “You have no tools and cannot run code. Compute the totals yourself.”
It was exact in 10 of 10 in the first run and 9 of 10 in the second. My guess of fewer than 7 is wrong. The one miss is also not about rounding. The model left out row 27 completely, 40 units at 2,462.10 with 15% discount, so net is lower by 83,711.40 and VAT by 19,253.63.
A median call took 42.9 seconds and $0.11, with about 4,000 thinking tokens. At 5 lines it was 13.6 seconds and $0.024. The engine does the same 40 line invoice in about 0.1 ms. All 100 calls of this measurement cost $5.71.
When I did not name the rule
My other guess was about rounding. I gave 10 invoices of 15 lines without the rule, only “to the grosz”. I expected the model to switch between rounding VAT per line, per rate or only at the end.
It did not switch. 18 of 20 answers were numbers and all 18 match VAT taken from the sum per rate. Per line and per rate give different totals in 4 of these 10 invoices. In those 4 the model chose per rate both times. In one answer it called this “standard Polish practice”. The other 2 answers were a plan and a program. The program was right too.
What I take from it
I was ready to write that the model cannot count an invoice. It can, in 19 of 20 cases at 40 lines when told to count. Its code was right every time. A wrong total that still adds up gets past my first check. A tool result that nobody ran looks like proof and is not. So my totals still come from the Decimal code.
What I did not check
I tested 1 model with 2 runs per invoice and the invoices are synthetic. There were no credit notes or foreign currency and nothing scanned from paper. The model never got a real tool, so I know its programs are right but not what it does with a real result. I also did not set the thinking budget myself and did not read the thinking, so I do not know where row 27 got lost.