What breaks when an AI reads a bank statement
Eighteen months of running a language model against real books at a small-business fintech. A list of the things that actually went wrong, with dates, because I kept notes.
We shipped the classifier in April 2025. A bank line comes in from QuickBooks, the model picks an account from that client's chart of accounts, an accountant looks at it and clicks approve or fixes it. I remember thinking the hard part was going to be the prompt. It wasn't. I don't think I touched the prompt in a meaningful way after June. What I touched, over and over, for a year and a half, was everything around it.
Some background so you know who's talking. I wrote trading systems for a long time, then went and got a financial planner certification (CFP, 2015, mostly out of curiosity about where the money went after my code moved it), and now I'm the senior engineer on the accounting side of a fintech in New York. So when something is off in a ledger I tend to feel it before I can explain it, which is useful, and also means I sometimes trust the feeling too much. Both of those show up below.
The one time the model was actually wrong
October 9th this year. The model returned an account_id that did not exist in the client's chart of accounts. Short, well formed, looked exactly like the real ones. We had a length check on that field. It passed. If the row had gone through, we'd have posted a purchase to an account that nobody had, and QuickBooks would have told us so, eventually, in a background job, in a way nobody was watching. The fix is one line: if the id isn't in the list we sent the model, it's an error. One line, and I'm a little embarrassed it took until October.
That is the only incident in eighteen months I'd honestly file under "the model made it up". The rest is us.
Everything else
The expensive one. Around September accountants started seeing lines marked "Deleted in QuickBooks" that were sitting right there in QuickBooks, not deleted. We blamed sync timing for a while. Then the provider. Then I read the sync code properly, which I should have done first. The sync built its set of "lines we saw this run" from the transactions it had already filtered. So anything a filter dropped (a deposit with no payee, a line that failed validation, anything on page two when we'd stopped reading at page one because the API's totalCount underreports) was treated as gone. And a month that never got re-synced stayed deleted forever. There was no model anywhere near this bug. It was a plain list-building mistake, and it had been quietly deleting rows since the beginning.
Fix merged September 30th: record presence from the raw payload before any filter runs, and if the fetch was incomplete, mark nothing. There's still a small residue of deletions I can't explain. My current guess is unstable page order, because the queries have no ORDER BY, but I haven't proved it, and I'm writing that down here so I don't forget it's a guess.
Then the statement PDFs. A bank statement is a weird file. The header and the bank's address are selectable text; the table with the actual numbers is often an image somebody pasted in. Our first extractor asked "did we get any text?", got yes, and handed the model a page of labels with no amounts. The model, being a model, returned a perfectly shaped JSON with nothing financial in it. Nobody noticed for a bit because it looked right. Now we use pypdf across every page, reject image-only files outright instead of OCRing them, and refuse to send anything to the model unless there are at least five numbers on the page that aren't years. Five is arbitrary. I know. It's still a sparse check, and I don't believe a cheap check exists that proves a table came through whole. The thing I'd do with more time is look at each page's modality separately and keep the text and the rendered image as two witnesses, and when they disagree on a number, stop and ask a human. We're not there.
Approval vs. posting. This one I got wrong conceptually, not just in code. An accountant approves a row. The row leaves the queue. A green toast says done. What we had actually done was write "approved" in our own database. The write to QuickBooks happens later, grouped by transaction, in a worker, and the worker can fail for reasons that have nothing to do with the approval: a stale object, an account that went inactive, a payload QuickBooks stopped accepting. So for a while a row could disappear from the queue, fail quietly, and show up nowhere the accountant was looking. We gave that a name, sync conflict, attached the provider's error and a note a person can read, and wrote down the contract that approved rows stay visible until the ledger confirms them. (As of September 9th that contract was still a document and not fully code. I'm being specific about that on purpose.)
Repairs, briefly. On July 7th a batch of rows got rejected by mistake and I wanted to put them back. The obvious fix is one UPDATE, set them all to pending review. Except each row had come from a different queue (AI review, accountant review, client review) and each queue has a paired "source" column saying who last touched it. A pending-AI-review row with source = accountant is a contradiction the system can't reason about, and I had nearly written it. So: each row goes back to the queue its own changelog says it left, status and source move together, the repair writes a new audit entry and leaves the old rejection in place, and you dry-run it with counts first. If the count surprises you, you don't understand the data yet. I didn't, the first time.
And the sandbox. August 20th. We were designing bank reconciliation on top of an open-banking provider, and in the sandbox every transaction carried a running balance, which made the design easy. Then we connected a real account. Zero of twenty four real transactions had runningBalanceAmount. The sandbox's sample statement had a closing balance that matched the account, which cheered me up for about an hour, until I noticed the PDF itself was dated 2017 and the metadata said 2026. So nothing in that sandbox could validate the thing I was building. The design shrank to: store the statement, store the ending balance if we have one, compare only when the provider actually gives us the data, and make "unavailable" a real state in the UI instead of a null.
Oh, and the thing that's probably obvious to accountants and wasn't to me: QuickBooks hands us eight transaction types and only three of them (purchases, bills, deposits) can be classified at all. Invoices come through as read-only. Journal entries, transfers and payments we only ever see inside reports. So "the AI reads your books" is, in practice, "the AI reads about a third of your books, the third that's mostly spending". I say that to people now before anything else.
What I'd tell someone starting this next week, in no particular order: validate every model output against the tenant's own records, because a well-formed id can be invented; never infer that a thing is gone from a list you already filtered; treat "we got text" and "we got the numbers" as two different events, per page; show the local decision and the provider's answer as two states, a toast is not a posting; and keep a set of real, reviewed lines and score every model or prompt change against it in CI. Ours has blocked more regressions than code review has. Also assume the sandbox is wrong about the exact thing you care about, and check with one real account before you design around it.
None of this wanted a bigger model. I suspect that's the uncomfortable bit for people who'd like AI accounting to be a model problem, because model problems get solved by someone else next quarter. These were data problems, and data problems are yours.
If you're running a model against real books and you've hit something that isn't on this list, I'd genuinely like to hear it. My list is eighteen months long and I'm sure it's not done.