← Zum BlogBack to the blog

"Autonomy on a leash: where human oversight belongs in agentic AI"

·4 Min. Lesezeitmin read·EN

Künstliche IntelligenzComplianceProduktstrategie

Agentic AI changes the question. A copilot drafts and waits; an agent acts — it books, sends, updates, escalates. The moment software stops proposing and starts doing, “human oversight” stops being a review step and becomes an architecture problem. The EU AI Act already frames it this way: for high-risk systems, oversight measures must be “commensurate with the risks, level of autonomy and context of use” (Regulation (EU) 2024/1689, Article 14). Autonomy and oversight are designed together, or not at all.

Why the approval gate at the end fails

The instinct is to bolt a human onto the end: the agent does its work, a person signs off. It feels safe. It isn’t. Route every action through one approval queue and you get one of two failure modes. Either the human becomes a rubber stamp — clicking “approve” faster than they can read, because the queue never empties — or the queue becomes the bottleneck the agent was meant to remove, and the team quietly turns oversight off to ship faster.

Both are the same mistake: treating oversight as a gate stuck to the end instead of a property designed into the system. Effective oversight, the Act notes, has to let a person correctly interpret the system’s output and decide to disregard, override or reverse it. You cannot retrofit that into a black box after the fact.

Oversight you add at the end is theatre. Oversight you design in is control.

Autonomy as a design decision, not a default

The useful move is to make autonomy explicit and graded, per action — not one global “human in the loop” switch. Three levels cover most cases:

  • Assisted — the agent proposes, a human commits. Right for irreversible, high-blast-radius actions: payments, external communication, deletions.
  • Supervised — the agent acts within tight bounds; a human is notified and can reverse within a window. Right for medium-stakes, reversible work.
  • Delegated — the agent acts autonomously; humans audit samples after the fact. Right for high-volume, low-stakes, well-understood tasks.

The design work is mapping each capability to a level before launch, with explicit triggers for when an action moves up a level — an unusual amount, low model confidence, a new counterparty. That is what “commensurate with the level of autonomy” means in practice: the more autonomy an action has, the tighter and more specific its intervention points — not a vague promise that “a human is watching.”

Make oversight measurable

An oversight design you cannot measure is a hope, not a control. The NIST AI Risk Management Framework organises exactly this under its Measure and Manage functions, with Govern as the cross-cutting spine (NIST AI 100-1). Translated to an agent, three numbers tell you whether the leash is real:

  • Escalation rate — how often the agent hands control back. Trending to zero on a high-autonomy action is a warning, not a win: the criteria are too loose, or people have stopped looking.
  • Override precision — when humans do step in, how often were they right to? Low precision means the escalations are noise, and fatigue is coming.
  • Reversibility — for supervised actions, what share were caught and reversed inside the window? Near zero means the window is decorative.

None of this works without a record. The Act’s automatic record-keeping obligation (Article 12) — logging events over the system’s lifetime — is not bureaucratic overhead; it is the substrate that makes both oversight and accountability possible. The OECD puts the same idea plainly: AI actors need the “capacity for human agency and oversight,” and stay accountable for the system’s proper functioning (OECD AI Principles).

The product takeaway

Agentic AI doesn’t remove the human; it changes where the human belongs. Not at a gate at the end, drowning in approvals — but upstream, in the design: deciding how much autonomy each action earns, what pulls it back under supervision, and which numbers prove the arrangement is holding. Get that right and autonomy is an asset you can defend to a regulator and an auditor alike. Get it wrong and you have a fast agent, an absent human, and no answer to the only question that matters when something breaks: who was actually in control?

Sources

Lass uns sprechenLet’s talk Alle BeiträgeAll posts