2 weeks after ChatGPT came out, the first use I wanted to try was taking the amount out of a payment document. I wanted to see what a failure looks like inside a pipeline, because nobody reads the answers there and only the checks see them. ChatGPT has no API yet and I cannot see its tokens. So I took Flan-T5, a model from Google that I can run on my own machine.
The setup is boring on purpose. A small HTTP server wraps the model and answers with a generated_text field, like the Hugging Face Inference API. It returns 500 only if Python throws. My client sends the document and a question, plus the line “Answer with the amount only” at the end. Then it checks what I would check for any internal service: status is 200, the body is JSON, the answer parses as an amount. I wrote these 3 checks first without thinking. Only later I understood that a wrong number passes all 3.
Versions, because the numbers depend on them: Python 3.9, torch 1.13.1 on CPU, transformers 4.25.1, greedy decoding. Flan-T5 base has 250M parameters and large has 780M, both with the weights published on 21 October. The documents are 600 synthetic ones from a script with a fixed seed, so I know every right answer. There are 6 kinds with 100 of each. These are an invoice, an invoice with an unpaid previous balance, a transfer confirmation, a receipt, a refund email and a payment reminder. Half of the documents write numbers as PLN 1,284.50 and half the European way, 1 284,50 PLN.
The dashboard
Base made 2,000 calls. Each document got the main question and the same question in other words. 4 kinds of documents also got 2 questions about other fields, which I need for a check later. All 2,000 calls came back 200 and every answer parsed. A Grafana panel for this service would show 0 errors.
161 of the 600 amounts were wrong.
The first wrong answer I opened was a receipt with TOTAL 437,78 EUR and the model answered 43778 EUR. My first idea was that my parser had eaten the comma, then that the tokenizer drops it on decode. I checked both before I believed it was the model. The tokenizer splits the input as ▁4, 37, ,, 78, ▁EUR and the model generated ▁4, 37, 78, ▁EUR. The comma was its second choice at that step, with a probability of 0.12 against 0.79 for 78. So the amount becomes 100 times larger. This happened 52 times on the right number and 10 more times on another number from the document. All 52 are in the format with the decimal comma.
The bigger class is harder to see. 68 wrong answers are another amount from the document, written as a normal amount, just not the one I asked for. The reminder reads “Invoice 52081 dated 2022-05-02 for PLN 2,501.80 is overdue. We received PLN 500.36 from you on 2022-09-21. The remaining PLN 2,001.44 is due…” Base answered with the original invoice amount in 73 of 100 reminders, 10 of them with the comma lost as well. 1 more reminder was wrong in another way. I thought it simply takes the first amount. But in the invoice with a previous balance the amount to pay is the last line. Base got that one right 87 times of 100. So I do not know what decides it. 27 are text copied from the document, mostly receipt lines, where my parser took the first number. The last 4 have digits mangled in some other way.
The checks I could add without labels
Before the run I wrote down what I expected, so I could not adjust it later. I expected the check “is this amount in the document” to catch less than half of the wrong answers. I expected asking the question twice in different words to catch more. Neither idea is mine. Wang et al. used agreement between answers in March and in July Kadavath et al. studied how well a model knows when it is right.
My first document check had a bug in the definition. It matched any run of digits, so an item count from a receipt line passed and the check caught 66 of 161. When I made it match only amounts with 2 decimals, it caught 91, more than half, so for base my expectation did not hold. It cannot raise a false alarm, since a right answer is always in the document. A strict format check wants a single number with groups of 3 digits and exactly 2 decimals. It catches almost the same answers, 91 of 161, with 2 false alarms on 439 right ones.
Asking twice did worse than I expected. It caught 53 wrong answers and flagged 105 of the 439 right ones. Base often gives 2 different answers and one of them is right. The model’s own confidence did better, but not enough to use alone. I took the mean log probability of the answer tokens and flagged the 20% least confident answers. That caught 67 wrong ones and 53 right ones. The AUROC is 0.76, so a random wrong answer is less confident than a random right one about 3 times in 4.
Arithmetic looked like the winner at first. In 4 kinds of documents the fields add up: subtotal plus VAT is the total, the invoice total plus the old balance is the amount to pay, the refund plus the fee is the original charge, paid plus remaining is the invoice amount. I ask the model for the other 2 fields and add them up. Where it applies, this caught 103 of 103 wrong amounts and flagged 130 of 297 right ones. Then I looked at the invoices. Asked for the VAT, base never once gave the right number and returned the subtotal 67 times, so on invoices the check flags everything. Without invoices it caught 87 of 87 wrong amounts, with 46 false alarms on 213 right ones.
The bigger model
Large was much better, but 3 times slower. On 4 cores of a virtual machine with an AMD EPYC 7713 it takes 1.6 seconds per call against 0.5. It got 35 of 600 wrong, again all with 200 OK and all parsed. I expected fewer errors, but not that the cheap checks would mostly stop working. The document check caught 7 of 35 and the strict format 8. Asking twice caught 7 too, with 41 false alarms. The least confident 20% caught 14 with 106. 28 of the 35 wrong answers are other amounts from the document, so they look exactly like right answers.
Arithmetic still works where it applies: 21 of 21, with 8 false alarms in 379, because large got 791 of the 800 field answers right. Together with arithmetic, the document check and the strict format catch 25 of 35.
The other 10 are all the same mistake and base never made it. In the transfer confirmation large answered with “Balance after transfer” instead of the amount. It did so 11 times and the strict check rejected only 1 of them. Base had 5 errors on transfers too, but those were lost commas and mangled digits, which the checks see. My transfer template has no balance before the transfer, so no sum ties the amount to the other numbers. Asking twice flagged 5 of the 10 and the confidence 2. For the rest I would need a second source, like the bank’s own record of the transaction.
What I did not check
- ChatGPT and
text-davinci-003. I measured 2 small open models only and I do not know if a much larger model loses commas. I guess it does not, but a right number from the wrong line is the error I would expect to stay. - Real documents. My 6 templates are short and clean, so the rate per template describes my templates. Base getting 74 of 100 reminders wrong probably says more about how I wrote the reminder than about the model.
- Prompts. One wording of the instruction and one paraphrase of each question. Another instruction could change the reminder result a lot.
- Repeats. Greedy decoding only. Every number here is from one run. I reran 2 documents outside the server and got the same tokens, but I did not repeat the whole run.