A successful AI pilot proves something useful, but smaller than teams often want it to prove. It shows that one version of a system handled a defined set of tests, with a particular configuration, at a particular time. It does not by itself establish that the system is safe, reliable or worthwhile in the conditions where people will actually use it.
The practical mistake is not running a weak experiment. It is allowing a promising experiment to authorise its own deployment.
A real warning from an AI product claim
In January 2025, the US Federal Trade Commission finalised an order against DoNotPay concerning claims about its AI service. The order requires the company to stop making specified deceptive claims about the service's capabilities unless it has competent and reliable evidence to support them.
The important point is not that every AI tool resembles DoNotPay. It plainly does not. The narrower lesson is that a fluent product demonstration and a defensible capability claim are different things. If a claim matters to a buyer, user or regulator, the evidence has to match the claim being made.
That is especially easy to lose in a pilot. The people closest to an experiment know its boundaries. Everyone else sees the summary: accuracy improved, time fell, users liked it. By the time that summary reaches a budget meeting, the test conditions can disappear.
Keep the experiment attached to its conditions
A useful experiment record should answer six questions.
- What exact task was tested? “Customer support” is not a task. “Draft a reply to a delivery-status question using the current policy library” is.
- What was the baseline? Compare the system with the present method, not with an imagined version of manual work.
- Which system version and configuration ran? Model, prompt, retrieval source, permissions and tool access can all change the result.
- Which cases were included? Normal cases alone tell you very little about exceptions, ambiguity or adversarial input.
- What failed, and how expensive was review? A faster first draft can still create more total work if checking it is difficult.
- What decision does the evidence support? Reject, revise, repeat, sandbox, limited deployment or stop are all legitimate outcomes.
This is the reason to record the failed cases with the successful ones. A failure is not an embarrassment to tidy out of the slide. It is information about the boundary of the system.
Deployment is a new question
NIST's AI Risk Management Framework is voluntary guidance, not a deployment certificate. Its value here is structural. The framework separates governance, mapping, measurement and management, and calls for testing before deployment and while a system is operating. It also makes leadership responsibility, human oversight, documented limits and context-relevant evaluation explicit.
Turn that structure into a short deployment conversation:
- Who owns the decision to deploy?
- Which users, decisions and data are in scope?
- What must stay outside the system?
- Who reviews output before any material action?
- Which failure rate or failure type stops the trial?
- How will people report a problem or challenge an outcome?
- When will the evidence be reviewed again?
If nobody can answer those questions, the next step is not a larger rollout. The next step is to make the boundary readable.
A worked example
Imagine a team testing an AI assistant that classifies incoming supplier enquiries. In a 200-message sample it routes most routine messages correctly and saves time. That is encouraging.
Now inspect the conditions. The sample contains few messages in languages other than English. It excludes attachments. The test uses last month's supplier list. A human checks every classification, but that review time is not included in the saving. Two incorrect classifications involve time-sensitive payment changes.
The result is not “the assistant works” or “the assistant failed”. A proportionate decision might be:
Repeat the experiment with multilingual messages, attachments and payment-change cases. Keep the system read-only. Measure total review time. Do not automate routing for payment or account-detail changes.
That is a smaller claim and a better decision. It preserves the value of the pilot without borrowing certainty from it.
The next useful action
Take the last AI test your team described as successful. Write down the task, baseline, system version, test cases, observed failures, human review burden and current deployment boundary on one page. If any field is unknown, mark it unknown. Do not repair the record with a confident guess.
The AI Experiment Logbook is the related catalogue tool for keeping that record reproducible. If the question has moved beyond an experiment into permissions, oversight and release authority, inspect the AI Agent Readiness Kit as the next control layer.
Sources and transfer limits
- FTC case record: DoNotPay — used for the fact and status of the FTC order. It supports a specific US enforcement example; it does not establish that every AI capability claim is unlawful or determine the rules in another jurisdiction.
- NIST AI Risk Management Framework Core — used for the governance, evaluation, oversight and monitoring structure. The AI RMF is voluntary US guidance, is being revised, and is not legal advice, independent certification or proof that a particular deployment is safe.
Sources reviewed 7 September 2026. Recheck them, the applicable law and the live system before a material deployment decision.