Before an enterprise asks what an AI agent can do, it should define what the agent is allowed to do. This sounds obvious, yet many agent programmes begin with technical capability: tools, memory, retrieval, multi-agent orchestration and API access. Decision rights arrive later as governance.
That sequence is backwards. An agent that can act across systems is effectively a new operational actor. Like any other actor, it needs a role, permissions, escalation path, scope of authority and evidence obligations.
Autonomy is delegated authority
Traditional automation is usually deterministic. A workflow performs a predefined action when predefined conditions are met. Agentic systems are different because they can select among actions, sequence tools, interpret context and adapt their path. That flexibility is where the value comes from. It is also where authority becomes less obvious.
If an agent receives an application, identifies a likely duplicate, asks for missing evidence, changes priority and assigns the case to a specialist queue, it is already exercising operational discretion even if it never makes the final eligibility decision.
The architecture therefore needs a decision-rights model that distinguishes observation, interpretation, recommendation, preparation, execution and final decision.
Use consequence, reversibility and evidence to set the boundary
I would not define agent autonomy primarily by technical confidence. Three business characteristics matter more.
Consequence: what happens if the action is wrong? Mis-prioritising an internal queue is different from denying a payment. Reversibility: can the action be undone cleanly? Drafting a message is easier to reverse than sending it. Evidence: can the organisation reconstruct the basis for the action and determine whether the agent operated within policy?
A high-confidence model output can still be inappropriate for autonomous execution if the consequence is material and the action is difficult to reverse.
Build a decision-rights matrix, not a generic human-in-the-loop statement
“Human in the loop” is often used as a comfort phrase. It does not tell an engineering team where the loop is, what the human approves, what information they receive, or what happens when the human disagrees.
A useful matrix names the business action and assigns an authority level. For example: summarise a case — autonomous; classify document type — autonomous with sampling; identify a duplicate — recommend; change case priority — execute within defined thresholds; approve financial assistance — human decision; override statutory eligibility — authorised human only.
Once this exists, the technical design becomes much clearer. Tool permissions can match authority. Approval steps can be explicit. Monitoring can focus on material actions. Exceptions can route to the right role.
Agents need separation of duties too
Enterprises already understand separation of duties in finance, identity and change management. The same logic should be applied to agents. An agent that proposes a production code change does not necessarily need permission to deploy it. An agent that assesses a payment anomaly does not need authority to release the payment.
In some workflows, a second deterministic or AI-based verifier can challenge the first agent before a human sees the recommendation. NIST’s recent work on evaluation probes for agentic AI is directionally important because it focuses on grounding and machine-readable audit trails rather than assuming that a single model output is sufficient evidence.
Design escalation as a first-class capability
Good agents should know when they are outside their operating envelope. That requires explicit escalation conditions: conflicting evidence, missing authoritative data, policy ambiguity, confidence below threshold, unusual value, suspected fraud, user dispute or repeated tool failure.
Escalation is not failure. It is part of the operating model. A well-designed agent should reduce the volume of routine coordination while increasing the visibility of genuinely ambiguous work.
What the board or executive risk committee should be able to see
- Which business decisions are fully autonomous, partially autonomous or human-only?
- Which systems and tools can each agent access?
- What is the maximum consequence of an autonomous action?
- What evidence is retained for recommendations and actions?
- How are overrides, complaints and abnormal outcomes fed back into evaluation?
The important maturity shift is to stop describing autonomy as a feature of the AI and start describing it as an organisational delegation. Once framed that way, security, risk, legal, architecture and operations can all participate in the same decision.
Authority should be expressed as policy, not hidden in a prompt
One of the least mature patterns in agent design is to encode consequential authority inside natural-language instructions and assume the prompt is the control. Prompts are useful behavioural guidance; they are not a substitute for access control, transaction limits, workflow gates or segregation of duties. If an agent can create a payment, change an entitlement, publish information or alter production state, the permission to do so should be enforced outside the model as well.
I would define decision rights in a simple matrix: action, maximum consequence, required evidence, approval threshold, reversible or irreversible, permitted tools and escalation owner. That matrix can then drive technical controls. The model may recommend an action, but the API can enforce the transaction threshold. The model may assemble a case, but a workflow can require a human approval for a statutory decision. This keeps governance connected to architecture.
Monitor disagreement, not just accuracy
For recommendation agents, one of the most informative operational signals is human disagreement. How often do experienced users override the recommendation? Are overrides clustered around a particular case type, data source or policy rule? Does disagreement increase after a model or prompt change? A raw accuracy score can conceal these patterns. Override telemetry turns human oversight into a learning mechanism rather than a ceremonial approval step.
Know when to reduce autonomy
Autonomy is not a one-way maturity curve. If data quality deteriorates, a source becomes unavailable, regulation changes or the agent begins encountering unfamiliar cases, the system should be able to fall back from execute to recommend, or from recommend to retrieve-and-summarise. Designing that degraded mode up front is a sign that the agent is being treated as part of an operating system rather than as a feature demonstration.
