How we built an AI agent that proved itself before rejecting a single invoice

Written by Ayshvarya Laxmi K

AI Engineer · LinkedIn

Engineering series

Before we let an AI agent reject a single invoice, we made it prove itself. That idea shaped everything about the invoice validator we built for an Australian commercial property group: an agent that now checks every vendor invoice before it reaches an approver.

Reading an invoice sounds like the easy part. Getting it right every time, on documents that follow no template and against records that never quite agree, is the engineering problem. Getting a team that is accountable for every payment to trust the result is a harder one still.

Here's how we built it, how we proved it, and why we deliberately stopped short of letting it approve anything.

Why a ticked checklist stopped proving anything

The client's FM team approves thousands of vendor invoices a year. Every approval meant the same five manual checks:

• Billing entity
• Purchase order number
• Work order number
• Vendor details
• Invoice value

Each one had to be looked up against the purchase order, the vendor register or the work order. Done properly, that's a few minutes per invoice. Across a year, it adds up to a large part of someone's job.

Time wasn't the real issue, though. When we went through more than a year of the client's own invoice data, a bigger problem showed up: at that volume, a completed checklist no longer proved the checks had been done. Nobody was being careless. No team can sustain those lookups thousands of times a year.

So the brief wasn't to make approvals faster. It was to make the checks real again, without taking the decision away from the people who own it.

Teaching the agent to read any invoice

Vendor invoices arrive as PDFs in every layout imaginable. The purchase order number might sit in a header, a table or a footnote, so there's no fixed position to anchor to. Instead of hunting for fields by location, the agent reads all the text in the document and works out from context which value is the purchase order, which is the vendor's name and which is the site address.

Two readers and a tiebreaker

For a payment control, one model's reading isn't enough. So every invoice gets read twice:

• Two readers: Models from two different major providers read the invoice in parallel.
• A comparison: When their readings agree, the values move on.
• A tiebreaker: When they differ, a third model settles it, and its reading is final.

A single misreading never reaches a check unchallenged.

When the same vendor goes by different names

Reading the text is only half the job. The harder half is matching it to the client's records. The way a client stores a vendor's company name or a site's address is rarely the way the vendor writes it. One side abbreviates, the other spells it out, and street names get shortened in ways no exact-match rule would catch.

So the agent matches on meaning. It decides whether two differently written names refer to the same vendor or the same site, and treats a genuine match as a match, not a failure. That judgment, more than the reading itself, is where the models earn their place.

Letting the client draw the line

Not every mismatch deserves the same response, and what counts as critical is the client's call, not ours. Their team shaped the rules with us in a working session, and the rules fall into two kinds:

• Hard gates cover anything tied directly to money: the purchase order number, the work order number, the invoice amount and the vendor's tax ID. Failing one returns the invoice to the vendor.
• Soft gates cover administrative details, such as the site name or the business name a vendor uses day to day. Failing one flags the invoice for the approver to weigh.

The purchase order is why money sits on the hard side. It sets the upper limit for the work, so without a valid one there's no way to tell whether you're paying too much or too little.

Where the line sits varies more than you'd expect. I'm working with a client right now who treats a missing postal code in a site address as critical, while for others the address is purely administrative. So the gates are configured per client, never hard-coded.

Proving it before it touched a payment

We didn't want the first sign of a problem to be a wrongly rejected invoice. So the agent had to clear two stages before it was allowed to act:

• The client's own sandbox. Their team tested the agent themselves, and some gates were reclassified on what they found.
• Advisory mode in production. The agent checked every live invoice and blocked none.

In advisory mode, it answered to two judges:

• Our team verified every flag on every invoice by hand, which is how we measured accuracy.
• The client's FM team kept approving invoices as they always had, with the agent's findings beside them, judging whether it caught what mattered and missed nothing.

Their verdict was practical: the information they used to hunt for was already on the page, and the summary made every approval easier. The only errors came on the first day. We traced them to their root, fixed them the same day, and the agent hasn't been wrong since.

Before we let AI reject an invoice, we made it prove itself.

Ayshvarya

Only then did we switch it on.

Why we stopped at rejection

Our first design went further: clean invoices would approve themselves. The client pushed back, and they were right to. Their team carries responsibility for every payment, and a payment depends on more than whether an invoice passes its checks.

Auto-rejection was a different question. An invoice that fails a hard gate would be rejected by the FM anyway. Making them open it, review it and send it back only costs them time. So we drew the line there:

• Hard gate failures go straight back to the vendor, automatically.
• Everything else still reaches the FM, pre-checked, for a person to approve.

The approver keeps the decision. Their queue is just shorter, and every invoice in it has already been checked.

Giving vendors a fair way back

An agent that can reject needs a fair way back. A vendor who believes a rejection is wrong can raise a dispute with a reason, and from there:

• The dispute skips the agent entirely and goes straight to the FM.
• The FM either approves the invoice or upholds the rejection.
• If the rejection stands, the vendor corrects and resubmits. A dispute can't be raised again and again in a loop.

Every dispute outcome is recorded, and every run is logged on the FM team's dashboard. Together, they give the client an independent, ongoing read on how the agent is performing, measured by the people who own the decision.

# two models read the invoice independently
first  = reader_a.extract(invoice)
second = reader_b.extract(invoice)
values = first if first == second else tiebreaker.extract(invoice)

# match by meaning against the client's records, then apply their gates
results = [gate.check(values, records) for gate in client_gates]

if any(r.failed and r.gate.hard for r in results):
    return_to_vendor(invoice, results)       # a dispute skips the agent, straight to the FM
else:
    send_to_approver(invoice, flags=results) # pre-checked; a person approves

What earned the agent its authority

The challenge was never building an agent that could read an invoice. It was building one that a team accountable for every payment would trust to act. That trust came from choices that are small on their own:

• Two independent readings and a tiebreaker
• Matching by meaning, not spelling
• Gates the client defines
• Evaluation by two judges, before any authority
• The right to reject, but never to approve

None of it is visible to the vendor or the approver. The invoice simply arrives checked.

The approver still keeps the decision. They just no longer have to wonder whether the checks were done.

Written by Ayshvarya Laxmi K, with the Facilio engineering team.