What to check before you hire an AI automation vendor

Start with your own number, not their demo

Before you take a single vendor call, write down what the manual version of this work costs you today. Count the hours per week, name the person doing it, and note what gets missed when they are out sick or buried in the busy season. If you run a phone-heavy service business, what answering the phone all day actually costs walks through how to price that out. Most owner-operators have never done that math, which is exactly why a vendor's demo looks so good: it is being compared to nothing.

The same test applies whether you run a self storage facility, a wedding venue, or a pet boarding operation. The vendor questions below do not change by industry. Only the workflow being replaced does.

Ask what they have kept running, not what they can build

Thinklytics, an AI workflow automation consultancy, built a fourteen-question framework for exactly this problem and groups the questions into four sets: data integration, agent observability, permission model, and pricing transparency. Their first and most useful question asks the vendor to show a workflow they shipped that is still running twelve months later, with a real owner inside the client's business. Anyone can demo a prototype. Few can point to one that survived a compliance review, an API change, and a staff turnover.

Thinklytics scores each answer from zero to two across all fourteen questions and recommends walking away from any vendor scoring below eighteen out of twenty-eight. A vendor who can only describe pilots, never a survivor, has told you something important without meaning to.

Borrow the vendor's own five questions, minus the pitch

Iffel International, itself a firm selling AI automation assessments, publishes five questions worth borrowing even though the article ends by pointing back at their own service. Can the vendor show a documented outcome from a business in your specific sector? Do they assess your existing systems before proposing a fix, or do they pitch their preferred product and figure out the fit afterward? Can they deploy in 30 days, or is the timeline measured in quarters? Do they measure success in time recovered and revenue captured, or in features and automation rates? Is their pricing built for a business at your revenue level, or is it enterprise pricing wearing a small business landing page?

Read past the sales page and the underlying test holds up: a vendor who studies your systems before proposing anything is a different animal from one selling a product that is looking for a problem to solve.

Make them define their own words

Ask a vendor where they draw the line between automation and AI agents. Thinklytics lists this as their second framework question, because too many vendors answer "everything is an agent now," and that is marketing language, not an answer. A useful answer draws an actual line: deterministic workflows for repeat volume work, agents for the judgment calls that branch, and a stated test for which side a given task falls on.

In a Gartner peer community discussion on vendor evaluation, one automation leader in healthcare and biotech lists six labels vendors use almost interchangeably: chatbots, assistants, co-pilots, agentic systems, AI agents, and expert agents. You do not need to memorize that list. You need the vendor to tell you, for your specific task, which one they are actually selling you.

Ask about the boring stuff: data, exits, and price shocks

In the same Gartner discussion, a CIO in education said the hardest deal breaker in their process is not speed or accuracy. It is whether the vendor can guarantee where the data is stored, and whether the business can exit the contract if the vendor's own future looks shaky. Smaller AI vendors raise this question by default: a two-person studio and a large funded company carry different survival odds, and your contract should say what happens to your data and your workflow if the vendor does not make it to next year.

A chief supply chain officer in government, elsewhere in that discussion, described a vendor whose pricing jumped fifty percent once the tool was already running inside their operation. Ask for the pricing model in writing before you sign, not after the workflow depends on it. A vendor who hesitates on that question has usually already answered it.

What we would want a vendor to admit, including us

We build and operate this kind of system ourselves. Right now that includes 197 scheduled jobs across 78 live integrations, inside 30 projects in production, and seven languages of content. That scale does not make us the right fit for every business. It means we have hit most of the failure modes described above, from our own side of the vendor table.

Two limits worth naming plainly. We take a small number of engagements at a time, so if a vendor tells you they can start next week regardless of scope, ask what they are deprioritizing to make room for you. And we do not run a fixed price list: pricing is set per engagement against the value the system creates. That is not an answer either, on its own. Ask any vendor, us included, to show the math behind that number rather than hand you a figure and expect trust.

The number to bring to every vendor call

Before your next vendor call, write down three numbers: what the manual process costs per month, how long you are willing to wait to see it working, and the price increase that would make you cancel. Bring all three into the conversation instead of leaving them in your head. A vendor worth hiring answers directly. One who redirects to the demo wants you thinking about features, not the math you just did.