How to know when an automation is ready to run unsupervised

The 2-week trap

There's a pattern we see on almost every automation rollout. A new workflow goes live, runs cleanly for 10–14 days, and someone on the team says "looks good, let's take it off manual review." Two weeks later, something breaks quietly — a record gets skipped, a notification fires to the wrong person, an edge case that wasn't in the test data does something unexpected — and nobody catches it for days.

The automation didn't break because it was bad. It broke because the team graduated it before it had earned unsupervised status.

This applies equally to marketing automation, ops workflows, and anything touching customer-facing data. The stakes vary; the pattern doesn't.

What "ready to run unsupervised" actually means

Unsupervised doesn't mean unmonitored. It means the automation can operate without a human approving each output — but there's still a system that notices when something goes wrong and an owner who responds.

We use a short checklist before we pull oversight from any workflow:

1. Error rate is below your defined threshold

First, you need a threshold. Most teams don't have one, which means they eyeball it. Pick a number before you go live: for example, fewer than 2% of runs produce an unexpected output. After 200+ runs, check whether you're inside that number. Don't graduate an automation based on 15 successful runs.

2. Every failure mode has a fallback

Map your failure modes before you launch, not after. What happens when the upstream data is missing? What happens when an API times out? What happens when a record matches two conditions instead of one?

For each failure mode, there should be a defined fallback: route to a human queue, send an alert, skip and log, or halt. "It shouldn't happen" is not a fallback.

3. An owner is assigned and reachable

An automation without an owner is a ticking clock. The owner doesn't need to watch it daily — they need to be the person who gets paged when the error rate spikes, who knows the workflow well enough to diagnose quickly, and who has the access to fix or pause it.

"The ops team" is not an owner. A named person is.

4. Alerts are set at the run level, not the outcome level

Most teams set alerts on the obvious outcomes: "send me an email if a step fails." That catches overt errors. It misses quiet failures — automations that complete successfully but produce wrong outputs because the logic ran on bad input.

Better alert design monitors run volume (is it running when it should be?), output distribution (is the ratio of outcomes shifting?), and latency (is it taking longer than usual?). Those signals catch problems before a customer or stakeholder does.

5. You've stress-tested the edge cases, not just the happy path

Test data almost always represents the happy path. Real data doesn't. Before removing oversight, run the workflow against historical records that include exceptions: duplicate entries, missing fields, records that were manually edited, anything that looked weird in the last 6 months.

If the automation handles those cleanly, you have more confidence. If it doesn't, you've found your next fix.

A note on AI agents specifically

Everything above applies to rule-based automation. For AI agents — workflows where a model is making judgment calls, drafting outputs, or routing decisions — we hold a higher bar.

AI agents should almost never be fully unsupervised on consequential tasks in the first 90 days. The failure modes are harder to enumerate upfront because the model can behave differently on inputs you didn't anticipate. We prefer a "supervised autonomy" model: the agent runs, a human spot-checks 10% of outputs on a rolling basis, and the threshold for removing that spot-check is much higher than for deterministic automation.

This isn't a knock on AI agents — they're genuinely useful. It's just honest about where the risk surface is different. You can see how we approach this in our AI agents work and in some of the examples on our case studies page.

The practical version of this checklist

If you want to put this into practice without a lot of process overhead, make it a one-page sign-off document. Before any automation moves from supervised to unsupervised:

• Error rate target: _____%
• Runs completed in supervised mode: _____
• Failure modes documented: yes / no
• Fallback for each failure mode: defined / not defined
• Named owner: _____
• Alert types configured: run volume / output distribution / latency / failure
• Edge case testing completed: yes / no

One page. Sign it. Keep it in your ops wiki. When something goes wrong at 2am, you'll know exactly what was checked and what wasn't.

If you want to walk through your specific workflows and see which ones are actually ready, book a call.