Reading a Payslip Is Harder Than It Looks

Here is a payslip. It says the employee earns $4,120 this fortnight. Now tell me their annual income.

You cannot, yet. Some of that $4,120 is base pay and some is overtime, and overtime is treated differently. There might be an allowance in there that ends when the project does. The year-to-date figure at the bottom looks like the reliable number until you notice the employee started in October, so annualising it across a full financial year understates them by a third. The package might be quoted inclusive of superannuation, in which case a slice of it has to come out, at the rate that applied to that pay period rather than the rate today. And if there was a leave payout in this cycle, the year-to-date total is inflated by money that will never appear again.

Six decisions, one piece of paper. Now do it again for the second applicant, and again for the bank statements you are cross-checking against, and then write the note explaining which figure you used and why, because that note is what a lender and an auditor will read.

We built a scanner that does the reading and the arithmetic for a residential lending group. It does not decide.

Why a Payslip Resists Being Read

The manual version of this job is not hard in any single step. It is hard because it is a long chain of small judgements where any one link can quietly break the answer.

Fifteen to thirty minutes per applicant is a fair estimate for a clean file with one payslip and a bank statement. An hour is normal for two applicants with several documents and a super-inclusive package. None of that is client-facing work. It produces no advice and wins no business. It is transcription plus arithmetic, performed by someone expensive enough to be trusted with the judgement at the end of it.

Worse, the output of all that effort was often a single number in a form field. The reasoning that produced it lived in the assessor's head, and the working was not written down anywhere. When the figure was later questioned, the reconstruction cost nearly as much as the original assessment.

The Model Reads, the Code Decides

The obvious way to build this is to hand the document to a language model and ask it for an income figure. We did not do that, and the reason is the whole design.

The model's job is narrow: read the document and report what is on it. Employer, dates, pay frequency, each component of pay separated out, year-to-date totals, superannuation, the ABN. It reports observations, not conclusions.

Everything downstream is ordinary deterministic code, and it is the larger half of the system by a distance. The single biggest module is the derivation layer, at over nine hundred lines, and it contains no model calls at all. It groups documents by employer, drops superseded scans, isolates base pay from variable components, annualises from the correct start date, strips superannuation at the rate that applied to that period, and excludes one-off items that will not recur.

The split matters because of what a broker has to do with the answer. Arithmetic that a person will act on and later defend has to be reproducible, inspectable and identical every time. A language model is excellent at reading a badly-scanned payslip and poor at being the same calculator twice. So it reads, and the code does the maths.

The same principle governs the document recommendations. When the system suggests requesting a second payslip or a contract, that suggestion comes from a deterministic rule, not from the model. If software is going to tell a broker to go back to a client and ask for something, the reason had better be one you can point at.

Every Figure Carries Its Receipt

The rule that shaped the extraction contract is that no figure may appear without evidence.

Every extracted value carries the page number it came from and the verbatim line of text that backs it. Not a paraphrase. The actual line. If the model cannot find a line to cite, the instruction is to return nothing and lower its confidence, rather than produce a plausible number.

This turns out to be the single most useful thing in the whole system, and not for the reason we expected. It was built for auditability, so a broker could click a figure and see where it came from. What it actually does is make hallucination visible. A fabricated number has no verbatim line behind it, or has one that does not contain it, and both failures are obvious the moment a human glances at the card. The citation is not just the audit trail. It is the detector.

On top of the extraction, the code runs cross-checks that use the document's own internal arithmetic against itself. Employer superannuation as a proportion of pay, tested against the legislated minimum for that period. Implied pay cycles derived from the superannuation totals, compared against the calendar. The same year-to-date figure annualised two different ways, flagged when the two disagree beyond tolerance. None of these change a number. They raise a plain-English flag, because the point is to tell a human where to look.

The Safety Feature That Broke Every Scan

The best story in this build is the day the guardrail became the outage.

The extraction was originally enforced using structured output, where you hand the provider a JSON Schema and it constrains the model's decoding so the response cannot come back in the wrong shape. It is the responsible choice. It is the thing you reach for precisely because you do not want to trust a model's formatting.

Then we shipped a version that expanded what the scanner captured: per-allowance breakdowns with their own categories and permanence risk, one-off year-to-date items, employer superannuation totals, the employer ABN. All additive, all optional, all sensible.

Every scan failed. Not degraded, not occasionally wrong. Every single one, immediately, with a 400 before a single token was generated.

Constrained decoding works by compiling your schema into a state machine that restricts which tokens may be sampled next. That machine has a size limit. The new nesting, an array of allowance breakdowns with two enums inside an array of acceptance methods, pushed the compiled schema past the provider's serving limit. The API rejected the request outright.

Sit with the shape of that failure for a second. The mechanism that exists to guarantee correctness was the single point of failure. It failed at request time, not response time, so no amount of defensive parsing, retry logic or downstream validation would have caught anything. There was nothing to catch. And the trigger was not a bad model or a bad document. It was our schema getting more thorough.

The fix inverted the trust model. We stopped sending the schema. The output contract was already described in full in the system prompt, in prose, including every legal enum value, so we asked for JSON and enforced the shape ourselves on the way in.

That enforcement is deliberately, almost aggressively forgiving. Every field has a fallback, so the parser cannot throw. Enum values are upper-cased and trimmed before matching, and fall back to a neutral value rather than failing. Status synonyms are mapped, so anything containing "fail" or "error" lands on failed and the various flavours of success all land on completed. If the model returns a bare number where the contract expects a value with its citation attached, the number gets wrapped rather than rejected. Confidence is clamped into range.

The trade is explicit and worth stating plainly. The provider no longer guarantees the shape, so the application has to tolerate any shape and coerce it. In exchange the extraction contract can be as complex as the problem actually is, written in prose a human can read and argue with, and it is not bounded by a compiler limit inside somebody else's serving stack. The old schema is still in the file, never sent, kept as the canonical description of the shape with a comment explaining why it is dead.

The same fix cleaned up a bug the outage had been masking: pay frequency now gets inferred from the pay period span when the model cannot determine it, because the cross-checks all divide by pays per year and had been silently doing nothing whenever frequency came back unknown.

Refusing Is a Feature

A scan below a confidence of 0.85 does not flow through to the recommended figure. It goes to a review queue where a person approves or rejects it, and approving is recorded against their name. A rejection supersedes the scan and requires a written reason.

The prompt also refuses by design. An employer letter comes back as failed with "requires human read", because the useful content of an employer letter is prose and a person should read it. Self-employed income, rental, benefits, pensions and foreign income all return a named out-of-scope refusal rather than a guess. The system knows what it is for, and says so when handed something else.

The gate is set so the system fails toward holding a figure back rather than quietly submitting a wrong one. In a regulated workflow that is the only direction worth failing in.

There is one more rule that only makes sense once you have watched a real file go wrong. Household income is summed from figures a human has committed, per applicant, never from raw extractions. An earlier version added a payslip-derived figure to a bank-derived figure for the same person and double-counted one wage seen two ways. The fix was not better matching. It was refusing to add anything a person had not signed off first.

By the Numbers

This pipeline replaces the recurring cost of the income verification function:

• A loan writer (~$95,000 AUD/year, Hays Salary Guide FY26, Sydney typical) who would otherwise open every income document, transcribe the figures, and perform the annualisation arithmetic by hand for each applicant.
• A credit analyst, roughly half a role (~$85,000 AUD/year, Hays Salary Guide FY26, Lending Operations table) who would otherwise separate base pay from variable components, resolve superannuation treatment, cross-check against bank credits, and write the note justifying the figure used.

Operational savings breakdown:

• $137,500 AUD in annual savings, on base salary figures excluding superannuation.
• 550 dev hours avoided against a traditional build. Roughly 4,500 lines across the pipeline, the derivation engine, the routes and the reconciliation interface, plus a versioned extraction contract that took real iteration to get right. This is an estimate of comparable build scope, not a measurement.
• 1.5 FTEs replaced.
• Automation type: ai-integration.

The tests concentrate on the deterministic layer, which is the correct place for them. The model's reading is exercised against real documents rather than fixtures, because a unit test cannot tell you whether a payslip from an employer nobody has seen before will parse.

Where to Put the Guardrails

The instinct with a language model in a regulated workflow is to constrain the model harder. Tighter schema, stricter output, more validation at the boundary. That instinct is what produced an outage in this system, because the constraint mechanism had a limit nobody had told us about and it failed closed on the entire feature at once.

What actually held up was cheaper and duller. Let the model do the one thing it is genuinely better at than code, which is reading a document a human wrote badly. Take its output as untrusted input, because that is what it is. Do every calculation in ordinary code that runs the same way twice. Make every figure carry the line it came from, so a wrong answer looks wrong. And leave the decision with the person whose name goes on the file.

The number that reaches the lender is still chosen by a human. What changed is that they now spend their time checking a figure instead of assembling one.