The pilot worked and the rollout never happened

The pilot was scoped to succeed, not to be representative

Most pilots are designed by people who want them to work, which is understandable and is the origin of the problem. The cases are hand-picked from the clean end of the distribution. The data used is an extract somebody tidied. The team involved is the one that volunteered. And there is usually one unusually capable person available on a message thread for the duration, resolving anything odd within the hour.

None of those conditions exist in production. The real distribution includes the awkward cases that consume most of the handling time. The real data has the fields people stopped filling in three years ago. The real users did not volunteer. And the capable person has moved to the next engagement.

A representative pilot is built differently. It takes a random sample rather than a chosen one, including the awkward cases, and it runs against the real data with its real gaps. It is used by a team that did not ask for it. And somebody writes down, before it starts, what result would mean stop. A pilot with no failing condition defined in advance is a demonstration, and a demonstration is a legitimate thing to buy as long as everyone knows that is what it is.

• A random sample of cases, including the ones nobody wants to show
• The real data, with the fields that are empty in practice left empty
• At least one team that did not volunteer
• A stated result that would mean stop, agreed before anything runs
• A record of every case the pilot could not handle and why

A pilot with no failing condition defined in advance is a demonstration. That is a fine thing to buy, as long as everyone knows that is what it is.

Nobody owns it the day the pilot team leaves

The pilot has a project manager, a vendor contact and a sponsor. Production needs an owner, which is a different thing. An owner is the person whose job it is to decide what happens when the workflow meets a case nobody anticipated, to approve a change to the process itself rather than to the tool, and to settle a disagreement between two departments about what the system should do.

Without one, the pattern is consistent. The system runs. It produces something slightly wrong for a category of case. Nobody has the authority to change the process that produced the wrongness, so somebody builds a manual workaround around the automation. Six months later the workaround is the process, and the automation is decoration nobody is willing to switch off.

The test is simple and uncomfortable. Name the person. Then check that changing the process is inside their job description rather than a favour they would have to ask for. If the honest answer is that this belongs to a committee, treat that as the finding: a committee can approve a pilot and cannot own a system.

It was never connected to the system of record

Pilots are frequently built beside the business rather than inside it. Data is exported, processed somewhere else, and the output is pasted back by a person. That is a reasonable way to test whether the processing works and a terrible foundation for scaling, because the part that was skipped is the part that is hard.

Extending a pilot like that is not extending anything. It is starting the integration project that was deferred, usually with a budget that assumed the pilot was most of the work. Everything that makes integration difficult is still ahead: authentication, field mapping, what happens when the same record is edited in both places, what the system does when the other side is down for an hour, and who is told when a write fails silently.

The correction is not to build the full integration during the pilot. It is to prove one write, once, into the real system, in the real environment, even if the rest of the pilot runs on exports. A single successful round trip tells you more about the cost of production than a hundred processed records do, and it surfaces the credential and permission conversation while it is still cheap.

No baseline, so there is no verdict

Ask what the number was before, and a surprising share of pilots cannot answer. There is a sense that things are better. There is a satisfied user quote. There is a chart with no comparable earlier version. So the decision about whether to scale gets made on impressions, which means it gets made on who is more persuasive in the room.

A baseline is not complicated and it has to be taken before the build, because afterwards it cannot be recovered honestly. Measure the thing the project claims it will change, in the units the business already uses, over a period long enough to include a bad week. Handling time per case, cases per person per day, time from arrival to first response, error rate and rework, whichever the project is actually about.

Then measure the same thing the same way afterwards, and publish both, including the cases the system could not handle. A pilot that improved handling on the cases it accepted, while quietly returning the difficult ones to a queue for someone else, has not improved the process, and only a baseline that counts everything will show that.

• The number, measured before the build, over a period that includes a bad week
• The same measurement afterwards, taken the same way, by the same method
• The volume of cases the system declined or escalated, counted rather than excluded
• What the process cost to run, including the checking nobody puts on the plan

Questions to ask before you approve a pilot

These are the ones that change the answer. Ask them in the approval meeting rather than at the review, because every one of them is cheap to satisfy in advance and expensive to retrofit.

If the answers to the first three are unclear, the honest move is to spend two weeks fixing that rather than three months running a pilot whose result nobody will be able to interpret.

• What result from this pilot would mean we stop, and who agreed to that in advance
• Who owns this in production, by name, and can they change the process without asking permission
• What is the current number, how was it measured, and by whom
• Will this pilot write once into the real system of record, or only into a copy
• How were the test cases selected, and what proportion of real volume do they represent
• What happens to a case the system cannot handle, and who sees it
• What does the second department cost, once the first one is live
• Who runs this in month four, when the people who built it are elsewhere

What the engagement looks like, and when not to run one

Getting from pilot to production is normally a scoped project with an end, not a subscription. It has a shape: confirm the baseline, prove the integration, rebuild whatever the pilot faked, handle the exception path properly, then hand over documentation and access so the team can read and maintain the workflow. A support arrangement afterwards is a separate and much smaller thing, decided once the system exists rather than bundled in before anyone knows what it will need.

Three variables move the size of it more than anything else. How many systems the workflow touches and how many of those have a supported interface, since one application with no API can cost more than the rest of the work combined.

Whether the process is documented, or has to be mapped from how people actually do it, which is a phase of its own. And how many exceptions the process really has, because exception handling is usually most of the build and is almost always underestimated by the person who ran the pilot.

Anyone quoting production before walking the process end to end and listing every system it touches is quoting a sales number. The list is the estimate, and it is reasonable to ask to see it before agreeing to anything.

There are cases where a pilot is the wrong step entirely. If the process is going to be redesigned within the year, piloting automation of the current version is buying something with a known expiry. If the volume is low, the arithmetic may never support the build, and counting first is cheaper than discovering it after. And if the real blocker is that two departments disagree about who owns the work, a pilot will not resolve that. It will give the disagreement a new subject.

Common questions

Why do AI pilots fail to scale?

Four reasons account for most of it. The pilot was scoped to succeed rather than to be representative, nobody owns the system once the pilot team leaves, it was never connected to the system of record so scaling means starting the integration that was deferred, and no baseline exists so nobody can say whether it worked.

What makes a pilot representative?

A random sample of cases including the difficult ones, the real data with its real gaps, at least one team that did not volunteer to be involved, and a stated result agreed in advance that would mean stop. Without a defined failing condition it is a demonstration rather than a test.

Who should own an AI system in production?

A named person who can change the process itself, not only approve spending on it. If changing the process would require them to ask a committee, the project has a sponsor rather than an owner, and it will stall at the first disagreement about scope.

Should a pilot connect to our real systems?

It should prove at least one write into the real system of record, once, in the real environment, even if the rest of the pilot runs on exported data. Integration is the part that carries the unknown cost, and one successful round trip converts that unknown into a known before the production budget is set.

What should we measure before starting?

The number the project claims it will change, in the units the business already uses, taken over a period long enough to include a bad week. Handling time, volume per person, time to first response, error rate and rework are the usual candidates. Afterwards, measure the same way and count the cases the system refused.

When is a pilot the wrong step?

When the process is due to be redesigned within the year, when volume is low enough that the arithmetic never supports a build, or when the real blocker is that two departments disagree about who owns the work. A pilot does not settle an ownership dispute, it gives it a new subject.