How to evaluate an AI vendor when every answer is yes

Most vendor questionnaires sort nobody

Run a standard AI supplier questionnaire past five firms and you will get five sets of yes. Do you follow security best practice, yes. Is the solution scalable, yes. A question everybody can answer the same way has no sorting power, and a scoring sheet built from those produces a ranking of who writes the most confident prose.

A question worth asking has two properties. Answers to it differ between suppliers, and the answer can be checked rather than believed. Which usually means asking for an artefact instead of an assurance: show me the workflow, show me the contract clause, show me what happens when I revoke this credential right now.

There is also a category difference worth establishing early. Some suppliers build something for you and hand it over. Some resell a platform and configure it. Some rent you an output and keep the machinery. All three are legitimate purchases, and they behave completely differently in year two.

A question everybody answers the same way has no sorting power. Ask for the artefact rather than the assurance.

Ownership, meaning the output, the workflow and the instance

You own your data is the easiest sentence in this industry to say and the least useful, because it is almost never the thing in dispute. What matters is the layer above it: who owns the workflow logic, the instructions, the integrations, the configuration and any tuning done on your data. Those are the parts that took the time.

Ask where the system physically lives and in whose account. A build sitting inside the supplier's platform tenant, under their credentials, is a build you are renting regardless of what the ownership clause says, because possession decides what happens during a disagreement. Ask whether you can hold administrator access today.

Ask about anything derived from your data. If a supplier tunes a model, builds an index of your documents, or accumulates a corrected dataset from your team's feedback, that artefact was made out of your material, and its ownership should be written down before it exists rather than after it has become valuable.

And ask whether anything built for you gets reused for others. There is a reasonable answer, which is that generic components are reused and anything specific to you is not, and an unreasonable one, which is a clause allowing them to resell your process as a product.

• Who owns the workflow logic, instructions, integrations and configuration
• Whose account the system runs in, and whether you can hold administrator access
• Who owns anything derived from your data, including indexes and corrected datasets
• Whether components built for you may be reused or resold elsewhere

The day after the contract ends

This is the single most informative question in the set, because a supplier who has thought about it has usually thought about everything else, and one who has not is describing a relationship they assume will not end.

Ask it concretely rather than as a clause. If we gave notice today, what would we have in ninety days: a running system, an export, or records in a format nobody can use. Who holds the credentials to the integrations, and are those accounts in your name or ours. What happens to the platform subscription. How long does offboarding take, who does the work, and is it chargeable.

Then ask the harder version. Could our own team, or a supplier we choose, keep this running without you. If the honest answer is no, that is not automatically a reason to walk away, but it should be priced as a dependency rather than described as a partnership. Dependencies get repriced, and the moment they get repriced is the moment you cannot leave. Ask also what you would still be paying for in year three, and what for.

Legibility, hosting and who answers at three in the morning

Ask to see the actual workflow, not a diagram of it. A build your team can open, read and follow is a different asset from a service that produces results through a mechanism nobody on your side has seen. Legibility decides whether you can maintain something or need the person who built it.

Ask what runs where, and be specific. Which parts run on the supplier's infrastructure, which on yours, which on a third party platform, and what the failure of each would do to the others. Where hosting matters, the platform choice is not neutral: n8n can be self-hosted, so credentials and data can stay inside your own environment, while Make and Zapier cannot be. A firm committed to one platform will describe your problem in terms of what that platform does well.

Then ask about support in operational terms rather than in tiers. Who receives an alert when a workflow fails at three in the morning, and does it reach a person or a mailbox. What is the written commitment for acknowledgement and for resolution, and what happens when it is missed.

How many people at the supplier know your system well enough to work on it. Ask for the on-call arrangement as it exists, not as it would be described in a proposal. And ask what they will refuse to build, because a supplier who has never turned down a request usually does not have a view.

• Can our team open and read the workflow itself, today, without a walkthrough
• Which components run on whose infrastructure, and what one failure does to the rest
• Whether the platform can be self-hosted, if that matters to your data
• Who is alerted on failure, what the written commitment is, and how many people know the system
• What this supplier has refused to build, and why

Two questions Wobble would rather you did not ask

A vendor evaluation page that only lists questions the author answers well is a sales page wearing a checklist. Here are two where Wobble's answers are uncomfortable, with the answers.

The first: how long has your longest running system been in production, and can I speak to the person who operates it now. Wobble is a young firm, and its production systems are measured in months rather than years. There is a second constraint on the answer too. Wobble holds no written permission from any client to name them or describe their work, so it does not offer reference calls by default and will not supply an anonymised description detailed enough to identify anyone.

If your criteria require a decade of operating history and three referees, Wobble will score worse than a larger incumbent, and you should weigh that rather than accept a reassurance about it. What is offered instead is access to a working instance, a walkthrough of the actual logic, and the exit terms in writing before anything is signed.

The second: how many people would have to be unavailable before our system stopped being supported. In a small firm that number is small, and any small supplier claiming otherwise is describing a rota they do not staff.

Wobble's mitigation is structural rather than reassuring: the client holds the instance and the credentials, the workflow is documented and readable, and the build is made portable so another supplier or an internal team can take it over. That reduces the consequence without removing the risk. If your requirement is a follow the sun rota with named tiers and a contractual penalty, ask for it explicitly and check whether the answer describes a team that exists.

Ask both of every supplier you are considering, including the large ones. The large firm's answer to the second is usually a rota, and the follow-up worth asking there is how many people on it have ever seen your system.

• How long has your longest running system been live, and who operates it now
• How many people would have to be unavailable before support stopped

What good looks like on paper, and where this checklist misleads you

A responsible proposal has a shape you can recognise. A scoped project with a defined end and a named deliverable, rather than an open retainer for attention. A separate and smaller support arrangement decided after the system exists, because nobody can size support for something not yet built. Ownership and exit written down before work starts. And a price that arrives after somebody has walked the process end to end.

What moves the number is rarely the count of features. It is how many systems have to be connected and how well behaved they are, whether the process is documented or has to be mapped from how people actually work, and how many exceptions it really has, because exception handling is usually most of the build. Any supplier quoting before knowing those three is quoting a sales number, and that includes us.

Now the limit of this page. A checklist rewards the supplier who prepares for checklists. A well drilled sales team will answer all of the above smoothly, and a small firm doing excellent work may answer one badly because nobody has ever asked. Use the answers as prompts for a conversation rather than as a score.

None of these questions test whether the supplier can build the thing. Ownership, exit and support are contractual, and a firm can be flawless on all three and technically wrong about your problem. The check for that is different: give them a real process, with its real exceptions, and see whether the first response is a proposal or a set of questions about the exceptions.

Common questions

What should I ask an AI vendor that others will not?

Ask what happens the day after the contract ends, in specifics rather than clauses. What would you hold in ninety days, whose accounts the integrations sit in, whether your team could keep it running, and what you would still be paying for in year three.

What does owning the output actually mean?

More than owning your data, which is rarely in dispute. It means owning the workflow logic, the instructions, the integrations and the configuration, holding administrator access to the instance it runs in, and having written ownership of anything derived from your material such as an index or a corrected dataset.

How do I tell a builder from a reseller?

Ask to see the workflow itself rather than a diagram, and ask whether your team can open and read it. A build you can inspect and maintain behaves differently in year two from a service that produces results through a mechanism nobody on your side has seen.

Does it matter where the automation platform runs?

It matters when credentials or data need to stay inside your own environment. n8n can be self-hosted, so the instance and its credentials can sit in infrastructure you control. Make and Zapier cannot be. That is a factual difference between the tools, and the choice should follow your requirements rather than the supplier's preference.

What questions are awkward for Wobble to answer?

Two. Wobble is a young firm, so its longest running production systems are measured in months rather than years, and it holds no written client permission to give references. And it is a small team, so the number of people whose absence would affect support is small. Its answer to the second is that clients hold the instance and the workflow is readable and portable.

Can a vendor checklist replace a proper evaluation?

No. Checklists reward suppliers who prepare for checklists, and a well drilled sales team answers all of them smoothly. Use the answers to start a conversation, weight what you are shown over what you are told, and test capability separately with a real process and its real exceptions.