What happens when an autonomous agent makes a mistake?

Voice agents and the wrong refund policy

If you are asking how to make sure an AI voice agent does not give a customer the wrong refund policy, the answer is not a better script alone. The agent needs an authoritative source for policy text, a rule that blocks promises outside that source, and a handoff when the question is emotional or unusual. Air Canada's chatbot told a customer he could claim a bereavement fare after he had bought a full-price ticket; the tribunal held the airline liable for what the agent said, as CMSWire reported. Cursor's support agent invented a one-device policy that did not exist and cancellations followed before a human corrected it. Customers treat spoken policy as yours, not the model's.

What happens if a treasury AI agent makes a mistake

When money moves, a mistake is not a bad FAQ answer. Agentic payments sharpen liability: a Consumer Bankers Association and Davis Wright Tremaine white paper, covered by The Financial Brand, warns that Electronic Fund Transfer Act protections may not map cleanly once an agent initiates the transfer instead of a person. Customers will call the bank first when a payment goes wrong. A treasury or payment agent should hold transactions when confidence is low, route exceptions to a named approver, and log amount, counterparty, and approval state before anything irreversible runs.

Proving who was accountable after a costly mistake

If an AI agent made a costly mistake, accountability is decided from records, not from what the model intended. Pull the log for the action: inputs the agent saw, tools it called, outputs it sent, and whether a human gate was required or skipped. If no log exists, that absence is the finding. The business owns what the agent did on its behalf, which is why deployment should name who approves refunds, payments, and customer-facing commitments before automation runs at volume.

When an AI platform assigns the wrong business class code

When automated intake assigns the wrong business class code or a submission does not fit a standard pattern, the response should match how automated quote intake works for insurance brokers: binding authority stays with the producer, including judgment on a risk that does not fit a standard class code. A workflow can flag a premium far outside the expected range or a sublimit the client never asked for, route the comparison to the producer for a decision, and it should not bind a policy or choose a carrier on its own. When something is unusual, such as a class of business the agency has not placed before, the workflow should stop and hand the file to a person with a note about what is unusual, not guess at a mapping and send incomplete data to a carrier.

The mistake is real, not hypothetical

Air Canada found this out in front of a tribunal. Its chatbot told a grieving customer he could claim a bereavement fare after he had already bought a full-price ticket. That information was false. When the airline refused the refund and argued the chatbot was a separate entity responsible for its own words, the tribunal disagreed and ruled Air Canada liable, as CMSWire reported. The precedent held: the business owns what its agent says, not the software underneath it.

Cursor ran into a faster version of the same problem. Its AI support agent, nicknamed Sam, invented a policy limiting customers to one device per subscription and described it as a security measure, according to CMSWire. No such policy existed. The fabricated rule spread through developer forums within hours, and subscription cancellations followed before a person at Cursor could correct the record.

The legal ground under agents is still moving

Agentic payments sharpen the liability question further. A white paper from the Consumer Bankers Association and the law firm Davis Wright Tremaine, covered by The Financial Brand, warns that the consumer protections built into the Electronic Fund Transfer Act may not apply cleanly once an agent initiates the transaction instead of a person. Customers could end up liable for their own agent's mistakes, and banks, the paper notes, will be the first place people call when a payment goes wrong.

None of this is settled law yet, but the direction is clear enough to plan around now. If your agent books, orders, refunds, or replies on your behalf, the customer on the other end will treat its mistake as yours, not as a bug in someone else's software. That is not an argument against using agents. It is an argument for knowing, before you deploy one, exactly what it does the moment it is unsure.

Automation without a backstop makes it worse

Money spent has not guaranteed good outcomes. Organizations put $47 billion into AI initiatives in the first half of 2025, and 89% of that spending produced minimal returns, CMSWire reported, pointing to compliance complexity and organizational chaos as the more common cause than the underlying model. Customer service came up as one of the riskiest places to apply AI without enough care, because a bad answer lands on someone who is often already frustrated.

Results were better, per the same reporting, where AI supported a human agent rather than replacing one: faster responses and more empathy, with a person still accountable for the interaction. For a small operation, that argues for a specific split: hand the agent the repetitive volume, and keep a person as the one who steps in for anything upset or outside the normal pattern. Our breakdown of workflow automation versus agentic decision-making is a useful gut check for where that line sits in your own business.

Some mistakes cost more than a wrong answer

The Air Canada case was not really about a wrong fare policy. It was about a customer grieving a death, being told something false at the exact moment he needed the answer to be right, part of what CMSWire described as the emotional damage an AI mistake can cause. A pricing error is annoying. A false answer delivered to someone at a hard moment is the kind of mistake that ends up in front of a tribunal, or on social media, before it ends up on your desk.

This is the honest limit worth naming: an agent can be built to recognize that a situation is emotionally loaded or unusual, and to hand it to a person instead of answering on its own. It cannot replace the judgment that situation actually needs. Any vendor or system that claims otherwise is selling you the part that is easy to automate and hiding the part that is not.

What changes when the agent can catch itself

An agent built to operate, rather than just respond, does two things a simple chatbot does not: it checks the outcome of its own action, and it knows which situations sit outside its authority and stops to ask a person before they turn into a promise it cannot keep. That has to be a design decision from the start, not something added later. A refund that matches a hundred prior refunds can be approved automatically. A refund tied to the kind of situation that put Air Canada in front of a tribunal should not be.

Most agent failures actually happen at the seam between the agent's reasoning and the real system it acts on: a calendar, a payment processor, a CRM. We wrote about that seam separately in where AI agents fail when they touch real systems. If you are evaluating a vendor rather than building this yourself, our notes on how to evaluate an AI automation vendor cover the specific questions worth asking about that seam before you sign anything.

What to check this week

Pull up whatever agent already touches your customers, whether it answers the phone or processes refunds for you, and ask one question: what does it do the moment it is not sure? A good answer names a specific action, such as flagging the message or holding the transaction, and a specific place you can go look afterward.

Then go look at that log, if one exists. If it does not, that absence is the real finding, ahead of anything the agent might say next week.