Start by defining what the automation is actually supposed to do and what a wrong outcome looks like, before you build anything to measure it. For a marketing engine that writes and publishes content and runs outreach: did the right piece go out, in the right language, on the day it was supposed to, and did a follow-up actually fire when a lead didn't reply. For an operations system handling reporting, monitoring, and scheduling: did the report land with real numbers, did the alert fire when it should have, did nothing sit untouched for a week because a queue silently stalled. Ran and worked are two different questions, and most teams only check the first one.
The second question needs its own layer of tracking: completion rate per run, error and retry logs (a job that fails quietly on its third retry is more dangerous than one that crashes loudly and gets noticed), and an escalation count, meaning how often the system hands a decision to a person instead of guessing at it. That number matters in both directions. Too low and it's probably making calls it shouldn't. Too high and you've built something that still needs constant babysitting, which defeats the point. We run over 110 scheduled jobs across more than 40 live integrations at Precipitate, 88 separate systems in production, and at that count nobody is watching each one by eye. The monitoring has to be built into the system itself: logs, dashboards, and alerts that report on output, not just on whether a process started.
None of that tells you the output was good, only that the system executed. Someone still has to look at a sample of what it actually produced on a regular basis, especially anything a customer or lead will see, like an email, a reply, or a translated page. Skip that step and you're measuring that the automation ran, not that it worked, and that gap is exactly what lets a broken system look healthy on a dashboard for months. If you're deciding whether to automate something, pick the one output you already check by hand today and use that as your first real metric before you expand to anything else.