Human-in-the-Loop Is Not a Safety Strategy

Cody Larsen, M.D. is Chief Financial Officer and Chief Medical Officer at Vericence

This week I was working a night shift in a busy emergency department. A woman in her late fifties came in with vague abdominal discomfort and nausea. Our triage tool scored her low-acuity — her vitals were within range, her presentation was unremarkable, and the algorithm did exactly what it was designed to do. By the numbers, she could safely wait.

The triage nurse asked me to see her right away. Not because of anything in the data — because of how the patient was breathing, the way she was holding herself, a quality of unwellness that doesn't reduce to a vital sign. The nurse couldn't have told you her oxygen saturation, heart rate, or blood pressure predicted a problem. It didn't. She just knew something was not right.

The patient was septic. Bacteria had spread into her blood stream. A few hours of delay would have been the difference between a routine admission and an ICU bed — or worse. Fortunately, we intervened quickly and after a few days in the hospital she was back home feeling well with her family.

I think about patients like this whenever I hear an executive describe their AI deployment as safe because there's a human in the loop. Because the lesson of that night was not that the algorithm failed. The algorithm worked. The lesson was that the system worked — but only because the human in it had the authority to override a correct-looking output, the context to know something the data didn't capture, and a workplace culture where doing so was expected rather than punished.

Most enterprise AI deployments do not have all three of those things. Many have none of them.

Walk into any conference, RFP, or board meeting where AI is being discussed and you will hear the same reassurance: don't worry, there's a human in the loop. It has become the universal absolution. The phrase that ends the conversation about risk.

I want to suggest that, in most enterprise deployments, it's the wrong end of the conversation.

"Human-in-the-loop" describes a design pattern, not a safety property. A human can be in the loop and contribute nothing. A human can be in the loop and make outcomes worse. The questions that actually matter aren't whether a human is involved — they're where in the loop the human sits, what authority they have when they get there, and whether the system is built to make their judgment useful.

These three questions get answered well in a small number of AI implementations and badly in most of them.

Performative accountability vs. architectural accountability

There are two flavors of accountability showing up in enterprise AI right now.

Performative accountability is what most rollouts look like. A model produces an output — a claims decision, a loan disposition, a flagged transaction, a clinical recommendation. A human is asked to confirm it before the action goes through. On paper, this is human-in-the-loop. In practice, three things tend to be true:

1. The human is reviewing dozens or hundreds of outputs per shift.
2. The model's reasoning is opaque, or buried behind a confidence score.
3. The default action is approve, and pushing back costs the reviewer something — time, friction, a conversation with their manager.

The result is rubber-stamping with extra steps. The human exists to absorb liability, not to add judgment. When something goes wrong, the audit trail shows a sign-off — but the sign-off was never load-bearing.

Architectural accountability is different. It treats human judgment as a scarce, expensive resource — because it is — and designs the system around the decisions where that judgment is irreplaceable. Instead of asking a human to review everything, it asks the AI to do what AI does well (pattern recognition at scale) and routes the genuinely ambiguous cases to a human with the context, authority, and time to decide them well.

The difference is not philosophical. It shows up in outcomes.

Three questions to ask of any "human-in-the-loop" system

Before you describe a deployment as accountable, run it through three filters.

1. Is the human in the loop before or after the consequential action?

A reviewer who approves a decision before execution is in a fundamentally different position than one who reviews flagged decisions after the fact. Both patterns have their place. But "human-in-the-loop" gets used to describe the latter while implying the former. Be specific about which one you have.

2. Does the human have enough context — and enough time — to disagree?

A fraud analyst handling 400 alerts a shift cannot meaningfully evaluate any one of them. A claims adjuster with 90 seconds per case is approving the model. If your reviewer's throughput requirement is incompatible with deliberation, you don't have human review. You have human latency.

3. What happens when the human disagrees?

If overrides require manager approval, generate friction reports, or get measured against the model's accuracy — "override rates are too high this quarter" — you've built a system that punishes its own safety mechanism. Real accountability requires that disagreement be cheap and legible.

What this looks like across verticals

The pattern repeats. Same diagnosis, different industries.

In financial services, AI is excellent at scoring transactions against historical fraud patterns. It is worse at understanding why a customer who never travels suddenly has charges in three countries — and whether the right answer is to freeze the account, call them, or do nothing. The first is a volume problem the model should own. The second requires context the model does not have access to.

In insurance, models can triage claims with high accuracy. But the cases where the model is least confident — novel injury patterns, unusual documentation, edge-case policy interpretations — are exactly the cases where human judgment matters most. Many deployments route the easy cases to humans and let the model handle the hard ones, because that maximizes throughput. It also maximizes risk.

In healthcare, clinical decision support has been around for decades. The implementations that work treat the clinician as the decision-maker, with the AI surfacing patterns and prompting questions they might otherwise miss. The implementations that fail treat the clinician as a verifier of model outputs — and quietly degrade their judgment over time, because skill that isn't exercised atrophies.

In government services, eligibility algorithms have produced some of the most public AI failures of the past decade. In nearly every postmortem, there was a human in the loop. The human did not have the authority, the context, or the time to override.

The point

If your AI system is making consequential decisions in a regulated, high-stakes environment, you don't get to outsource accountability to the word "loop." You have to design for it.

That means deciding — explicitly, in writing — which decisions the model owns, which decisions the human owns, and what triggers a handoff between them. It means measuring override behavior as a signal of system health, not as a deviation to be minimized. It means resourcing the human side of the system the way you resource the model side: with training, tooling, capacity, and authority.

The companies that get this right won't have the most automated workflows. They'll have the most legible ones — systems where, when something goes wrong, you can answer the question "who decided this, and on what basis?" without ambiguity.

That's what accountability actually looks like.

The loop is the easy part.

Cody Larsen, M.D. is Chief Financial Officer and Chief Medical Officer at Vericence, an AI-driven engineering consultancy serving enterprises in regulated industries, and Associate Director of Ignite Emergency Physicians. He writes about the intersection of high-stakes operations, regulated industries, and technology.