Measuring the return on an AI system, honestly

Write the baseline down before anything is built

The most valuable hour in an implementation happens before it starts and costs nothing. Somebody writes down what the work looks like today.

Take the task being automated. Record how many times it happened last month, how long an instance took, how often it went wrong and had to be redone, and how long the customer waited. If last month is unavailable, count for two weeks. An imperfect number recorded beforehand beats a perfect number reconstructed afterwards, because a reconstruction is a memory, and memories drift toward whatever result arrived.

Write it somewhere the supplier can see and agree with, and date it. A baseline both sides signed is what stops the conversation in month four becoming two people describing different pasts.

The numbers worth tracking

Four categories cover almost everything a business actually cares about, and every one of them can be counted without a data team.

Convert time into value using the hourly cost you already know for the people doing the work. That arithmetic is yours and does not have to be published, but it does have to exist, because a saving expressed only in hours gets discounted by whoever signs the invoice.

• Time. Hours spent on the task across everybody who touches it, including the interruption around it rather than only the typing.
• Speed. Time to first response, time to a quote, time to resolution. These move first and are the easiest for anybody to verify without a report.
• Quality. How often the output had to be corrected, and how often the same customer had to ask twice.
• Leakage. The work that never happened at all: enquiries that got no reply, follow ups nobody made, records never created. This is usually the largest of the four and it stays invisible until somebody counts it.

Numbers that look like return and are not

Each of these appears in real reports, and each of them survives exactly one sceptical question.

The fourth is the most seductive, because the arithmetic behind it is genuinely correct. Saved minutes only become saved hours if the freed time went somewhere you can name. If the honest answer is that everybody is a little less rushed, that is a real benefit and it is not a return, and calling it one weakens every other number on the page.

• Messages handled by the system, which counts activity rather than outcome
• Automations built, which measures the supplier's output rather than your result
• Model calls or tokens consumed, which is a cost line being presented as an achievement
• Time saved per instance multiplied by every instance, without checking that anybody's week actually changed
• Satisfaction with the new system, collected from the people who chose it

Attribution, and how to be honest about it

Businesses do not hold still. You launched the system in the same quarter you hired somebody, changed a price and entered a busy season, and now a number has moved and three people have three explanations for it.

Three approaches make attribution defensible without a research department.

When you genuinely cannot separate the effects, say so. A qualified number that survives a sceptical question is worth more than a confident one that collapses the first time somebody pushes on it.

• Stagger the rollout. One branch, product line or channel gets it first, and the others become the comparison for a few weeks.
• Compare like periods. The same month last year beats the previous month if your business has any season to it.
• Watch the mechanism as well as the outcome. If the claim is that faster replies win more work, check that replies actually got faster and that fast replies convert differently. If only the outcome moved, something else moved it.

Reporting it so a non technical owner can check it

A measurement nobody can verify is a claim. The test is whether the owner can check a number without asking the person who produced it.

The sample check matters more than it looks. Numbers drift for reasons that never appear in a total, and reading ten real cases catches things no aggregate ever will.

• One page, the same shape every month, with the baseline printed at the top of it
• Each number defined in a sentence, including what it excludes
• A route from any total through to the individual records behind it
• A sample check: pick ten recent items at random and read what the system actually did
• Failures reported beside successes, because a report with no failures in it is a report nobody believes

Where a measurement will not stand up, and when to skip it

If there is no baseline, do not build one out of memory. Say plainly that the before was never recorded, start measuring now, and let the next period be the comparison. A reconstructed baseline is the easiest thing in any report to attack, and losing that argument costs you every other number on the page.

If the volume is small, figures bounce for reasons that have nothing to do with the system. A handful of instances a week cannot produce a trend, and presenting one invites a fair objection you cannot answer. Report the count, describe what changed, and leave out the ratio that would have looked impressive.

And some benefits are not worth measuring. If the real gain is that the owner stops answering messages at eleven at night, that is a legitimate reason to buy and it does not need a spreadsheet. Forcing it into the shape of a return usually produces a weaker argument than simply saying it out loud.

Who writes the baseline, and the cost that belongs in the same column

The most valuable hour happens before anybody is engaged, so do it in-house. Your own team writes the baseline, because they are the only people who can say what the work looked like in an ordinary week rather than in the week somebody was watching. A supplier offering to construct the baseline for you is offering to mark their own work, and the polite version of that offer is still the same offer.

There is a cost that belongs in the same column as the return and is nearly always left out of it. No price is published on this site, but what moves the number after the build is the running cost rather than the build itself: the platform bill, the model usage, and the hour a week somebody spends reading what the system did. Put all three into the arithmetic from the first month, because a return calculated against a build fee alone looks excellent and is wrong by the second quarter.

That hour a week is also where the stop lives. Money, pricing, anything published in your name and any serious complaint wait for human approval, and a case outside the rules hands it to a person. The count of those cases is one of the few numbers that measures a system honestly alongside the ones above, because it says how often the thing met something it was never built for. Moiz Khan owns automation architecture at Wobble and Haad owns growth and client solutions. Wobble works from Karachi, bills month to month and is answerable for what it operates across 25 engagements in six countries, which is the arrangement where a return that never arrives becomes visible quickly.

Common questions

How do you measure ROI on an AI system?

Record the before, then compare the same number afterwards. Count how often the task ran, how long each instance took, how often it needed redoing, and how long customers waited. Convert the time into value using the hourly cost you already know, and add the work that was previously being lost entirely rather than only the work that got faster.

What if we did not record a baseline?

Say so rather than reconstructing one from memory. Start measuring immediately and let the next period be the comparison. A baseline assembled after the fact is the first thing a sceptical reader attacks, and losing that argument discredits every other figure in the same report.

Which metrics actually matter?

Hours spent on the task, time to first response or resolution, how often output had to be corrected, and the work that never happened at all. That last category, enquiries with no reply and follow ups nobody made, is usually the biggest of the four and the one nobody counts until it is pointed out.

How do we know the AI caused the improvement?

Stagger the rollout so part of the business becomes a comparison, use the same period last year rather than last month if you have a season, and check the mechanism as well as the outcome. If the claim is that faster replies win more work, verify that replies got faster before crediting the change to them.

How long before we can measure anything?

Speed measures move within days, which is why a first workflow with a readable measure is worth preferring. Quality and leakage need a few weeks of real volume before they mean anything. Judging the whole system in the first fortnight usually measures novelty rather than value.

What should we do if the numbers do not move?

Stop before buying anything else and find out why. The usual cause is the choice of task rather than the quality of the build, and the second most usual is that nobody was given ownership, so the system quietly stopped matching the process. Both are recoverable and neither gets fixed by adding another tool.