What to check before you hire an AI automation vendor

Start with your own number, not their demo

Before you take a single vendor call, write down what the manual version of the work costs your business today. Count the hours per week, name the person doing the task, and note what gets missed when they are out sick or buried in the busy season. If you run a phone-heavy service business, what answering the phone all day actually costs walks through how to price that labor out. Most owner-operators have never done that baseline math, which is why a vendor's demo looks so good: the tool is being compared to nothing.

The same financial test applies whether you run a self storage facility, a wedding venue, or a pet boarding operation. The vendor evaluation questions below do not change by industry. Only the daily workflow being replaced changes.

Ask what they have kept running, not what they can build

Thinklytics, an AI workflow automation consultancy, built a fourteen-question framework for this evaluation problem and groups the questions into four sets: data integration, agent observability, permission model, and pricing transparency. Their first and most useful question asks the vendor to show a workflow they shipped that is still running twelve months later, with a real owner inside the client's business. Anyone can demo a fresh prototype. Few vendors can point to a system that survived a compliance review, an API change, and a staff turnover.

Thinklytics scores each vendor answer from zero to two across all fourteen questions and recommends walking away from any vendor scoring below eighteen out of twenty-eight. A vendor who can only describe pilots, and never a long-term survivor, has told you something important without meaning to.

Borrow the vendor's own five questions, minus the pitch

Iffel International, itself a firm selling AI automation assessments, publishes five questions worth borrowing even though their article ends by pointing back at their own service. Can the vendor show a documented outcome from a business in your specific sector? Do they assess your existing systems before proposing a fix, or do they pitch their preferred product and figure out the fit afterward? Can they deploy a working system in 30 days, or is the timeline measured in quarters? Do they measure success in time recovered and revenue captured, or in technical features and automation rates? Is their pricing built for a business at your revenue level, or is it enterprise pricing wearing a small business landing page?

Read past the sales pitch and the underlying operational test holds up: a vendor who studies your existing systems before proposing anything is a different animal from one selling a pre-built product that is looking for a problem to solve.

Make them define their own words

Ask every prospective vendor where they draw the line between standard automation and AI agents. Thinklytics lists this distinction as their second framework question, because too many vendors answer that "everything is an agent now," which is marketing language, not an answer. A useful answer draws a practical line: deterministic workflows for repeatable volume work, agents for the judgment calls that branch, and a stated test for which side a given task falls on.

In a Gartner peer community discussion on vendor evaluation, one automation leader in healthcare and biotech lists six labels vendors use almost interchangeably: chatbots, assistants, co-pilots, agentic systems, AI agents, and expert agents. You do not need to memorize that industry list. You just need the vendor to tell you plainly, for your specific task, which category they are actually selling you.

Ask about the boring stuff: data, exits, and price shocks

In that same Gartner discussion, a CIO in education said the hardest deal breaker in their purchasing process is not speed or model accuracy. It is whether the vendor can guarantee where customer data is stored, and whether the buyer can exit the contract if the vendor's own financial future looks shaky. Smaller AI vendors raise this question by default: a two-person studio and a large funded company carry different survival odds, and your contract should state what happens to your data and your workflow if the vendor does not make it to next year.

A chief supply chain officer in government, elsewhere in that discussion, described a vendor whose renewal pricing jumped fifty percent once the tool was already running inside their operation. Ask for the complete pricing model in writing before you sign, not after your workflow depends on it. A vendor who hesitates on that question has usually already answered it.

What we would want a vendor to admit, including us

We build and operate this kind of production system ourselves. Right now that footprint includes 230+ scheduled jobs across 60+ live integrations, inside 30+ projects in production, and seven languages of content. Operating at that scale does not make us the right fit for every business. It simply means we have hit most of the real-world failure modes described above, from our own side of the vendor table.

Two operational limits are worth naming plainly. We take on a small number of client engagements at a time, so if a vendor tells you they can start next week regardless of scope, ask what work they are deprioritizing to make room for you. And we do not publish a fixed price menu: pricing is set per engagement against the measurable business outcome of the work. That is not a complete answer on its own, either. Ask any vendor, us included, to show the concrete math behind their figure rather than hand you a quote and expect trust.

The number to bring to every vendor call

Before your next vendor call, write down three specific numbers: what the manual process costs your business per month, how long you are willing to wait to see the workflow working, and the renewal price increase that would make you cancel. Bring all three figures into the conversation instead of leaving them in your head. A vendor worth hiring answers those boundaries directly. One who redirects to the demo wants you thinking about software features, not the math you just did.