Since ChatGPT came out 2 weeks ago, my feed is full of screenshots where someone pastes a document into it and gets the fields back. A bank statement is the classic case for this. Every bank exports it in its own way. The usual answer is a parser with a regex for each bank, which breaks the day the bank changes the export. I have never measured how a model compares with such a parser. I assumed that a model would at least copy a date correctly. So the question for me was only how often it gets the amount wrong.
The setup
I took 4 statement layouts and generated 150 transactions in each with a script, seed 2022, so every right answer is known. Every transaction should become 3 fields: the date, the amount with a sign (money out is negative) and the counterparty. The layouts look like this:
A 14.06.2022 CARD PAYMENT MEDIA EXPERT 7734 GDANSK -248,22 18 848,37
B Date: 2022-06-20 Description: Transfer to Piotr Wisniewski, title: gift Amount: -2,327.19 PLN Balance: -5,575.15 PLN
C 12 Jun 2022 CARD PAYMENT BOOKING.COM 33.40 EUR 157.32 DR 25,180.67 CR
D 15/06/2022 Card payment 4.55 53,361.89
AUCHAN 5293 GDANSKThe last number in each line is the balance after the transaction. Card payments abroad also carry the amount in the original currency. I wrote the regex parser for A and B only, 34 lines of Python, before I looked at C and D. C and D play the bank that changed its export.
The model is Flan-T5 from Google, base (250M parameters) and large (780M), released in October. It runs on my machine, 4 cores of an AMD EPYC virtual machine, Python 3.9, torch 1.13, transformers 4.25.1, greedy decoding. Each transaction gets 4 questions: the date, the amount, the direction of the money and who the other party is. The prompt also has the line of the statement with the first and the last day of the period. A real page has it and it tells which date format the bank uses. Both methods get the transactions already cut into blocks, so splitting the page is not measured here.
I count a transaction as right only if all 3 fields are right. A wrong result is loud when it does not parse. That means the regex did not match or the model gave a date that is not a date. It is silent when it parses and is wrong.
The regexes
On A and B the parser got 293 of 300 transactions right. The 7 wrong ones were mine. In layout A the name and the transfer title run together in capitals: TRANSFER FROM JAN KOWALSKI LOAN BACK. I took the first 2 words as the name. For the salary from SOFTHOUSE SP. Z O.O. this gives SOFTHOUSE SP., without any error. I wrote that rule without thinking about companies.
On C and D the parser matched nothing, 300 of 300 loud. This was what I expected. It is also the usual argument for the model.
The model
Flan-T5 large got 91 of 600 transactions right. Base got 107, so the larger model was not better.
I expected mistakes in the amount, but the first thing that broke was the date. I spent some time thinking my date parser was broken before I looked at the raw answers. With the period line in the prompt, large answered with the first day of the period in 360 of 600 transactions. For a payment made on 14 June it gave 01.06.2022. So I removed the period line and asked again. Large got the date right in 291 of 600 and base in 567. Often the answer of large is not the date from the line: 01.10.2022 for 10.01.2022, 01/26/2022 in the American order for 26/01/2022. Sometimes the year was 2020. It got 148 of 150 right only in layout B, where the date was already written as 2022-06-20. When I asked it to copy the date exactly as written, it did so in 162 of 600. 143 of them were in B.
Large answered “out” for every incoming transfer except in layout D, where the line says Incoming transfer in words. It answered “in” for 33 of 129 incoming transfers, all 33 in D. Large still answered “out” on lines without a minus and on lines with CR. I thought the order of the options in my question was to blame. So I asked again with “in or out” instead of “out or in”. Large did not change a single answer. Base did depend on the order. It answered “out” in all 600 when “out” came first and “in” in 583 of 600 when “in” came first.
The amount without the sign was right in 423 of 600. In 114 of the wrong ones the model returned the balance, the last number in the line. In 22 it returned the amount in euros or dollars from the description. As a counterparty large often returned more than the name, like SPOTIFY INTERNET. 61 times it returned only the city. If I accept any answer that contains the right name, it gets 526 of 600. With that lenient match large would get 160 transactions right instead of 91.
Almost all of this is silent, because a balance parses as an amount just as well as the amount does. Of 509 wrong transactions of large, 32 were loud.
Regex first, model when it does not match
This is the setup I would actually build: the parser for the formats I know and the model only for lines it does not recognise. It got 335 of 600 right, more than each of the 2 alone. But 256 transactions came out wrong without any error. Apart from my 7 salary lines, all of them came from C and D, the 300 lines that had failed loudly before.
Then I did what I would do anyway when a bank changes its export: I wrote the regexes for C and D. That took 29 more lines. The parser got 582 of 600 right and 0 loud. The 18 wrong are the same salary bug, which I copied into layout C together with the rest of the code. It costs about 6 microseconds per transaction. Large needs a median of 1.2 seconds per question, about 5 seconds per transaction.
What I did not check
This is one small model that I can run myself. It is not the GPT-3 behind the OpenAI API and not ChatGPT, which has no API yet. Nothing larger was tested. I used one prompt per field without examples and did not train anything. Making the model output all fields at once in JSON is also something I did not try. The statements are synthetic and clean text, without PDF and OCR.
For now I keep the regex per bank. The model I would use only to suggest a parse for a line the parser does not know. A person checks it before it goes anywhere.