What an automation pilot must prove before you buy
By Precipitate · 16 November 2026

An automation pilot should prove that the system can complete a defined workflow, detect failures, and hand uncertain cases to a person. Speed alone is weak evidence. The buying decision should rest on repeatable results, visible records, controlled exceptions, and the work still left for your staff.
Define the decision before the pilot starts
A pilot needs a written buying decision. State the process under test, the records involved, the actions the system may take, and the conditions that would stop a purchase. Without that frame, a polished demonstration can pass while the operating problem remains untouched.
Start with the current workflow. Record where work arrives, who handles it, which tools they open, and where judgment enters. Our guide to what automation discovery should uncover explains this mapping in more detail. The pilot should test that real sequence, including its awkward edges. A tidy sample created for the pilot proves very little about a messy inbox, booking calendar, or customer database.
Establish a baseline your team can defend
Measure the manual process before the automation runs. Useful baseline measures include elapsed processing time, staff attention, rework, missed items, duplicate actions, and unresolved exceptions. Choose measures tied to the reason for the pilot. Login counts and dashboard visits do not show whether work was completed correctly.
Effective measurement needs a baseline before implementation, continuous tracking, and systematic analysis. Attribution can become unclear when other business conditions change. Keep a record of unusual demand, staff absence, policy changes, and tool outages during the pilot. That context prevents the system from receiving credit or blame for unrelated changes.
Test complete outcomes, not isolated steps
A successful run should end with a business outcome that can be checked. A lead-handling pilot might need to read an enquiry, identify missing details, update the correct record, send an approved response, schedule follow-up, and record what happened. Drafting a message proves only that the system can draft.
Write an acceptance rule for each outcome. The rule should identify the expected destination, required fields, permitted action, and evidence of completion. Then run ordinary work through the same path repeatedly. Include incomplete requests, duplicate records, conflicting details, unavailable tools, and customers who change their instructions. These cases reveal where AI agents fail in real systems, which is often where buying risk sits.
Require visible failure behavior
A pilot should produce evidence from failed runs, not hide them from the report. Ask what the system does when a record is missing, an integration rejects an update, or a request falls outside policy. Good behavior may mean retrying a safe action, pausing the workflow, or asking a person to decide. Guessing is not acceptable when the action affects money, schedules, customer promises, or permanent records.
Set boundaries before testing. Identify actions the system may complete alone, actions that need approval, and actions it must refuse. The pilot should show that those boundaries hold under pressure. It should also show how staff receive an escalation, what information accompanies it, and how the workflow resumes afterward. Limits on what an AI agent can do alone should be part of the acceptance criteria, not an informal promise.
Inspect the evidence left behind
Every meaningful action should leave a record that an owner or operator can understand. The record should show what entered the workflow, what the system concluded, which tool it used, what changed, and whether the result was checked. Sensitive internal reasoning does not need to appear. The operational facts do.
Review a sample of successful runs, failed runs, retries, and human escalations. Confirm that records connect each input to its final outcome. Check whether a duplicate message, wrong booking, or incorrect status change can be traced without searching across several tools. Our article on audit trails for AI agents covers the records needed for investigation and accountability. If the pilot cannot explain an action, unattended operation will be hard to trust.
Measure the burden that remains after launch
Automation can complete the target workflow while creating a new supervision job. Track how often staff must correct records, rewrite messages, clear stalled runs, or answer avoidable escalations. Include maintenance work such as updating rules when forms, policies, or connected tools change. This workload belongs in the buying decision.
The final pilot report should separate owned work, shared work, and human-only work. It should list unresolved failure modes and the response expected for each one. It should also state who will monitor the system, repair integrations, review incidents, and approve material changes after deployment. A system that requires constant checking has not proved that it can operate autonomously.
This week, choose a recurring workflow and inspect a normal batch of completed items. Record the outcome, exceptions, rework, staff intervention, and evidence left behind. Those observations are the starting acceptance criteria for any automation pilot you consider.
Sources
Want this answered for your own business?
Get a straight answer →