Since ChatGPT came out 2 weeks ago, the idea I hear most in fintech is to let a language model check documents. Take an invoice and check that the lines add up to the total. I had my own assumption about it and I never tested it. I thought reading is the hard part. The model has to find the right number next to the right name. Adding 4 numbers after that looked like the easy step.
ChatGPT has no API, so I cannot send it a few thousand invoices. I took a model I can run on my own machine and measured where it stops being right.
The setup
The model is Flan-T5 from Google, base (250M parameters) and large (780M), released in October. It runs on 4 cores of an AMD EPYC virtual machine, Python 3.9, torch 1.13, transformers 4.25.1, greedy decoding. One question takes about 0.4 s on base and 1.5 s on large.
A script generated 400 invoices with seed 2022, so every right answer is known. Each has 2 to 5 lines. Half of the invoices have whole amounts up to 99, like PLN 45. The other half have amounts with grosze up to 999.99, like PLN 412.35. An invoice looks like this:
Invoice No. 55049
Seller: Marek Nowak Consulting
cable ties: PLN 9
paper reams: PLN 68
drill bits: PLN 90
desk lamps: PLN 92I asked 6 kinds of questions about the same invoices. 2 of them only need reading: the amount of one line and the total when the invoice has a Total: line. 4 need a calculation. 2 of them work on 2 lines, the sum and the difference. The 3rd is the total when there is no total line. The 4th is quantity times price, for a line written as 6 x PLN 73.82. Each prompt is the document and the question, followed by Answer with the amount only. An answer is right only if it has exactly 1 number and that number is the right one.
Reading
Large read 1,393 of 1,413 line amounts right and 388 of 400 printed totals. Base was worse with 1,154 lines and 360 totals. It often did not stop after the amount and gave the next lines too.
The mistakes of large are not random. Of its 20 wrong line answers, 19 are a single wrong amount and 18 of those are the amount of the line above. 14 of the 19 are in invoices with 5 lines and whole amounts. All 12 wrong totals are 1,000 or more. These are also the only totals with a thousands separator. Large returned PLN 1,218.24 for 2,188.24 and PLN 30005.03 for 3,005.03. Totals under 1,000 it read right 228 of 228 times.
Calculating
Large got 6 of 1,600 calculation questions right. Base got 0. All 6 were on invoices with whole amounts, so even there it is 6 of 800. In 2 of the 6 the right sum was also printed as another line of the invoice, 79 and 91. Large returned that number and I cannot tell if it added or copied.
The wrong answers are what worried me more. For the total without a total line, large returned the amount of one of the lines in 201 of 400 cases. In 16 more it returned the invoice number. In the other 182 it returned a number that is not in the invoice. All of them look exactly like a total. The answer is 1 number with PLN in front of it and a parser accepts it without an error. For quantity times price it was different. Large copied the line back, 6 x PLN 73.82, in all 400 cases. For large that is the only kind of question where the failure is loud, because the answer has 2 numbers in it.
Asking it to show the steps
Flan-T5 was trained with chain of thought examples, so the fair next try was to add Let's think step by step. to the sum questions. Large then writes the calculation out. On the invoice above it answered:
The total amount of all items is PLN 9 + 68 + 90 + 92 = PLN 340. The answer: 340.The right total is 259. Here every number to the left of = is right and only the addition is wrong.
I ran this only on the first 200 invoices, 400 sum questions, because 1 answer takes about 8 seconds. Large wrote an expression in 398 of them. Its own final answer was right in 3. Then I took the expression it wrote and evaluated it in Python. 223 of 400 were right. For a moment I thought this was the fix, but the totals spoiled it. In the questions about the total without a total line, the expression had every line of the invoice in only 89 of 198 cases. With 2 lines it was 51 of 55, with 5 lines 8 of 43. In 99 of the other 109 the model left some lines out of the sum. In the rest it misread or repeated an amount.
Reading with the model, adding with code
So I split the work along that line. The model reads each line amount with one question per line and Python adds them up. Large then got 386 of 400 totals right, against 1 of 400 when it was asked for the total directly. Each of the 13 wrong totals has a misread line in it. For 1 invoice a line came back with 2 numbers, so the code refused to add it.
Base gave 211 right totals this way and 18 wrong ones. In the other 171 some line came back as PLN 9 paper reams 68 and the code could not add it. Asked for the total directly, base was loud like that 45 times and silent the other 355.
This is not my idea. A paper from last month, PAL by Gao and others, does the same thing in general. The model writes a program and the interpreter runs it. The models built for FinQA and TAT-QA, 2 datasets of such questions about financial reports from 2021, also let code compute the result.
Where I would put the line
For me the boundary is clearer now. If the number the model returns is printed in the document, I can at least check that it is there. It would not catch the line above, but it is something. If the number has to be calculated, Flan-T5 on these invoices almost never gets it right. It returns something with the right shape. A check that the answer is not a number from the document would catch 217 of the 399 wrong totals of large. The other 182 would still pass.
What I still do not have is a rule for the reading mistakes, the line above in long invoices and the totals over 1,000. They are rare and a parser takes them like any right answer.
What I did not check
- Larger models.
text-davinci-003from OpenAI has been available through the API for about 2 weeks. I wanted a model on my own machine for this run and did not try it. Bigger models may add 4 numbers correctly. - Real invoices. These are synthetic, in English, one line per item. There are no scans and no OCR errors.
- Other prompts. I used 1 wording per question and greedy decoding. Few shot examples could change the numbers, mostly for base.