How to test an AI vendor's claims without a CTO
A practical way to evaluate AI vendor claims when nobody on your team is technical, including the questions to ask and the number to judge by.
The problem with a good demo
An AI vendor demo is a rehearsed performance. The examples are chosen. The data is clean. The person driving it knows which questions the tool answers well. None of that is dishonest. It is a sales asset and it tells you very little about how the tool behaves on your own messy records.
Without an in-house technical lead you cannot read the code or audit the model. You can still run a useful evaluation. What you are testing is not the technology. You are testing whether the claimed work actually gets done on your inputs at a quality your team will accept.
What an honest evaluation costs
This does not cover a security audit or a legal review of the vendor contract. Both sit outside what a non-technical evaluation can tell you and both need a specialist.
The real cost is staff time rather than fees. Expect one person to spend several hours assembling test inputs and several more judging the outputs. Many vendors charge for a paid pilot and GST at 18% applies on service fees. Budget for the exit as well. If you load data into the tool during a trial you may have to pay again later to get it out in a usable format.
When this is not worth doing
Skip a structured evaluation when the tool is cheap and reversible. For a low monthly subscription that one person uses and can cancel in a click the evaluation costs more than the mistake. Buy it and watch it for a month.
Do not run this process if you have not written down the task yet. A business that cannot describe its current workflow in plain steps has no yardstick to test against. The exercise will produce opinions rather than evidence. Fix the process first.
Also skip it when the decision is already made for other reasons. If a parent company or a major client requires a named vendor then an evaluation is theatre and your time is better spent negotiating the contract.
How to run the test
Ask for the demo to be re-run on data you supply. A vendor who refuses has told you something useful. Send a sample that includes your awkward cases such as unusual formats or mixed-language records or entries with fields missing.
Hold back a second sample the vendor never sees and run that one yourself during the trial. Ask the vendor directly for three examples of work the tool handles badly. A vendor with no answer either does not know their product or will not say.
Then ask the questions a technical lead would ask. Where is the data stored. Is your content used to train anything. What happens to your data if you stop paying. Who is accountable when the output is wrong. Get those answers written into the contract rather than left in an email thread.
The number that tells the truth
The metric that matters is the correction rate. Of a hundred outputs how many did a person have to change before they were usable and how long did each fix take. A tool that is right four times in five still costs you time if finding the fifth means reading all five.
The flattering metric is accuracy quoted by the vendor on their own benchmark. It describes their test set and not your work. Time-saved estimates are equally soft when nobody measured the old process before the tool arrived.
The expensive mistake
Most businesses test the tool on the work it was obviously built for and then sign an annual contract on that basis. The cost appears later. Real workloads contain the odd case and the angry customer and the invoice in a format nobody anticipated. Those are the instances where a wrong answer is expensive and they are exactly the instances a demo avoids.
The second half of the mistake is the commitment length. An annual contract signed on a two day trial removes your only leverage. Pay monthly until the correction rate has stopped moving.
Re-test at every renewal
Tools in this category change quickly. A product that was weak six months ago may be adequate now and a product you rejected may have added the one capability you needed. The reverse happens too. A vendor can change the model underneath and the behaviour your team relied on shifts without notice.
Treat every renewal as a short fresh test rather than a formality. Keep the held-back sample from your first evaluation so the comparison means something. Contract terms and data protection obligations should be confirmed by a qualified professional for your own situation.
This guide explains how something works. It is not legal, tax or financial advice, and rules change. Confirm anything regulatory with a qualified professional before you act on it.