What I'm learning putting AI to work inside real organizations.

Before you automate it, ask what breaks if it's wrong

Daniel Cornelius · November 7, 2026 · 2 min read

Most teams decide how much to trust an AI agent by asking how accurate it is. That's a reasonable question, but it's the second one to ask.

The first is: what breaks if it's wrong?

I call this the blast radius. It's the size of the damage when the agent makes a mistake, and it varies far more between tasks than accuracy does.

Two mistakes that look similar and aren't

Take two agents that are both 95% accurate.

The first creates CRM records from inbound emails. When it's wrong, you get a duplicate record. Someone spots it, merges it, and moves on. The cost is small, the fix is quick, and the error is easy to reverse.

The second takes action on live customer accounts, say by cancelling a subscription based on a request in a support thread. When it's wrong, a customer who didn't ask to leave is suddenly gone. Someone has to notice, apologize, restore the account, and win back trust. The cost is large, the fix is slow, and some of the damage can't be undone.

Same accuracy. Very different consequences.

Two things to measure

When I look at a candidate for automation, I check two properties of a wrong answer:

  • Size. How bad is the worst realistic outcome?
  • Symmetry. Is the cost of a false positive about the same as the cost of a false negative, or is one far worse than the other?

A small, symmetric mistake is a good candidate to let run. A large, lopsided one needs isolation and a gate: a human confirming the action, or a hard limit on what the agent can touch.

SMALLSIZE OF THE WORST MISTAKELARGEEVENCOST SYMMETRYLOPSIDEDDuplicate CRM recordQuick fix, easy to reverseLet it run, watch the numbersCancelled subscriptionSlow fix, some damage can't be undoneIsolate it and add a gate

Match the gate to the blast radius, not to your gut.

Why gut feel fails

Gut feel tracks how impressive the agent seems, not how costly its mistakes are. A fluent, confident agent earns trust it hasn't earned. And a rarely-wrong agent can still be dangerous if its rare mistakes are expensive.

The reverse happens too. Teams put heavy review on low-risk tasks because the agent feels new, and then wonder why the automation isn't saving time.

A quick way to use this

Before building, write one line for each candidate:

  1. What is the worst realistic mistake this agent could make?
  2. Can we reverse it, and how fast?
  3. Who notices, and how soon?

If the answers are "minor, easily, quickly," let it run and watch the numbers. If they're "serious, no, eventually," don't skip the gate. Plan how it comes off later, once the measured accuracy clears the bar you set for that risk.

The gate shouldn't be permanent, and it shouldn't be arbitrary. It should be sized to what's at stake.

What's the highest-stakes task your team has handed to an agent so far? I'd like to hear how you decided how much oversight it needed.

Enjoyed this? Subscribe by email on Substack or via RSS. I post one note a week.

Discuss this post on Substack →