Designing a Statement OCR Pilot Whose Result Actually Transfers

A pilot is a prediction: it claims that performance on these documents forecasts performance on next year's. Most pilots are poor predictions because the sample was assembled by someone who wanted it to work — the clean PDFs, the two banks everybody uses, the pages that scanned nicely. The result is true and does not generalise.

Why the underlying problem is hard is a separate subject. This is about designing the test.

Sample by stratum, not by convenience

Draw the sample from actual intake over a defined period — a full month, including month-end — and stratify it deliberately rather than taking the first hundred files. The strata that matter are the ones where behaviour differs: issuing bank, native PDF against scan against phone photograph, language and script mix, statement length, and account type where a salaried current account and a business account look nothing alike.

Then do the thing nobody wants to do: include the documents someone would normally set aside. The photograph with a thumb over the corner. The scan at an angle. The statement from the bank you only see twice a year. Those are not edge cases; they are the population that defines your exception rate, and excluding them is how a pilot produces a number that flatters everyone and predicts nothing.

A useful discipline is to have the sample drawn by someone with no stake in the outcome, or drawn mechanically — every nth document received. It costs nothing and removes the largest source of bias in the whole exercise.

Ground truth is the expensive part, and there is no shortcut

You cannot measure accuracy without a verified answer to measure against, and producing that means a person transcribing statements by hand. This is tedious, it is the main cost of a serious pilot, and skipping it is why so many pilots report vendor-supplied figures instead of measured ones.

Two practices make it defensible. Have a subset double-keyed by two people independently, so you know your own error rate — if your ground truth is 99% accurate, you cannot credibly distinguish two systems that differ by half a percent. And write down the ambiguous decisions before you start: whether a wrapped description counts as one transaction, whether a reversal is one row or two, how an opening balance line is treated. Those definitions are where two honest people produce different "correct" answers, and settling them afterwards means settling them in whichever direction favours the result.

Measure the things that map to work

Character accuracy is the wrong unit because nobody consumes characters. Measure at the levels where a human has to intervene:

The last row is the one that gets ignored and matters most operationally. A system with 94% accuracy and confidence scores that reliably flag the wrong 6% is more useful than one with 97% accuracy and confidence that is uninformative — because the first can be operated with a review queue and the second requires checking everything. Measure it directly: sort by confidence and see whether the errors cluster at the bottom.

Set acceptance criteria before you see any results

Write the pass threshold down, per stratum, before the pilot runs. This is the single cheapest discipline available and the most commonly skipped, because criteria set afterwards are set to match whatever happened. "Field accuracy above X on native PDFs and above Y on phone photographs, with the reconciliation check passing on at least Z per cent" is a decision. "It looked good" is not.

Set the thresholds by working backwards from operational capacity: how many statements will you receive, how many exceptions can the team actually clear in a day, therefore what pass rate is required. That derivation gives you a number you can defend to a vendor and to your own management, and it reframes the question from "is this accurate?" to "can we run this?" — which is the question that matters.