How to Measure Whether an AI Agent Is Working: Accuracy, Exceptions, Hours Back

If you want to know whether an AI agent is working, agree a baseline before it goes live, then track three numbers every week: accuracy against a person's verdict, exception rate and hours returned to your team.

Too often, AI in FM is judged on a feeling, which is one of the most common reasons AI projects in FM stall. In this guide, we have tried to explain how to set the baseline, score each number and fix one when it slips.

Why do you need a baseline before an AI agent goes live?

Because without one, every result is an opinion. A baseline is a snapshot of how the work runs today, taken before the agent touches it.

We take three measures for every process an agent will run:

• Volume: how many requests, jobs, invoices or documents come through each week.
• Time per task: the minutes of human effort per item, from start to close.
• Error rate: how often today's process gets it wrong, from reassigned jobs to billing errors to missed renewals.

Two to four weeks of history is usually enough. Most of it already sits in your CMMS and your work order analytics. The rest we time with your team over a few days. Treat the result like any other set of facility management KPIs: same definitions every week, same owner, same report.

This is what the baseline looks like for each agent we run:

How do you score an AI agent's accuracy?

Against a person. For every decision the agent makes, an experienced member of your team gives a verdict: right, wrong, or a better option than they would have picked.

Step 1: Score everything in shadow mode first

For the first one or two weeks, the agent proposes and your team decides. Every proposal gets scored. This is how we take AI work order dispatch live, and it is the same for every other agent.

Step 2: Keep scoring a sample once it is live

After go-live, we keep scoring a sample every week. Higher-risk work, like invoices and compliance gates, gets a bigger sample than routine triage.

Step 3: Set the bar for each agent

Good looks different for each job. A voice AI for facility service desks that picks the wrong category now and then is easy to correct. An invoice check that misses a billing error costs real money. So we agree the bar per agent with your team, before go-live.

One rule we never bend: the agent's own confidence score is not its accuracy. Confidence decides what goes to a person. Accuracy is what a person says about the result.

What is an exception rate, and what should happen to exceptions?

An exception is any item the agent hands to a person instead of finishing itself. The exception rate is the share of items that end up there.

A high rate is not automatically bad. Early on it should be high, because the rules start cautious. What matters is why items become exceptions, so we tag every one with a cause:

• Rule: a gate you set on purpose, such as spend above a cost limit. These stay with a person by design.
• Data gap: missing or messy data, such as a vendor with no insurance on file. Fix the data and the exception goes away. This is the kind of gap AI vendor compliance tracking catches.
• Low confidence: the agent was not sure. These are the ones we tune.
• Out of scope: work the agent was never set up to do.

Track the rate by cause every week. Rule exceptions should stay steady. Data gaps and low-confidence exceptions should fall. In AI vendor coordination, for example, a vendor who stops replying after the last follow-up becomes an exception, with the whole thread attached for the coordinator.

How do you count hours back without overstating them?

Conservatively. If the number will not survive a finance review, it is not worth reporting.

We use one simple sum. Take the items the agent closed with no person involved, multiply by the baseline minutes per item, then take off the time your team spent on exceptions and reviews. Here is a worked example for AI invoice validation, with illustrative numbers:

Two more rules keep the number honest. Do not count time nobody spent before, like checks that never happened. Those are quality gains, not hours. And keep money out of it. Hours back is the operating number. The business case turns it into money later.

What do FM Ops Engineers change when a number slips?

Rarely the agent itself. Most slips trace back to one of four causes, and each has a clear fix.

Accuracy drops: check the data first

A new site, a new vendor or a changed SLA often explains it. We update the data or the rule, then re-score.

Exceptions climb: read the causes

If data gaps are growing, we fix the source. If low-confidence items are growing, we tune the thresholds or add examples from your team's corrections.

Hours back stall: look at volume and scope

Sometimes the agent is accurate but sees too little work. We widen what it covers, one category at a time.

The team stops trusting it: show the record

Every action has a timestamp and a reason. We walk through the decisions with your team and fix the ones they disagree with.

How does measuring an AI agent compare with how AI is usually judged?

Most AI projects get one review at the end of the pilot. An agent should get a report card every week.

How does Facilio measure its AI agents for you?

We measure every agent we run against a baseline we agree with you before go-live. Our agents are part of our AI agents for facility operations, and our FM Ops Engineers own the numbers with your team. We report accuracy, exceptions and hours back on a regular cadence agreed with you, usually weekly. When a number slips, we change the data, rule or threshold behind it.

This is the kind of scorecard your team sees:

Why it does more than the way AI tools are measured today

What FM teams see on Facilio

Results our customers have reported on the platform, from published case studies.

You should never have to take an agent on trust. Every week, you should be able to see what it got right, what it handed back and what it gave your team.

The bottom line: is measuring an AI agent every week worth it?

Yes. Once the baseline is set, it takes about an hour a week, and it is the only way to know whether an agent earns its place. It also builds the trust your team needs before you hand the agent more work.

Here is a simple first step. Pick the process you want an agent on first. Write down this week's volume, the minutes each item takes and how many went wrong. That is your baseline, and everything else builds on it.

Frequently asked questions about measuring AI agents