When Should AI Agents Require Human Approval?

Key takeaways

1. Classify the action, not the model: the same agent may safely read a record but need approval to edit or send it.
2. Default unknown tools to higher risk: missing policy is not permission.
3. Show a concrete preview: approvers need destination, scope, side effects, data disclosed, and rollback path.
4. Revalidate after approval: permissions, prices, records, and tool arguments may change while an action waits.
5. Keep enforcement outside the prompt: use authorization middleware, scoped credentials, tool allowlists, limits, and logs.

How should an AI agent action be classified?

Start with the effect a tool can cause. A practical policy separates four classes: read-only actions; reversible internal writes; consequential external or privileged actions; and destructive or irreversible actions. OWASP’s AI Agent Security Cheat Sheet recommends least privilege, explicit risk classification, confirmation for sensitive tools, and human-in-the-loop controls.

This table is a starting policy, not a universal standard. The same operation can change class with context. Reading a public catalogue is different from reading a medical record; sending an internal test message is different from contacting a customer.

Where should the approval boundary sit?

Place approval immediately before the consequential tool call, after the agent has prepared validated arguments but before any side effect occurs. That keeps the preview specific and prevents a broad early approval from authorizing a later action the person never saw.

The approval request should include the tool, destination, proposed payload, evidence used, sensitive fields disclosed, expected side effects, cost or scope limit, expiry time, and rollback path. Secrets should be redacted. The user should be able to edit, approve, or reject without exposing credentials.

Why must the agent revalidate after approval?

An approval is a time-bound authorization for a specific proposal. Before execution, verify that the proposal has not changed, the approver still has authority, credentials remain scoped, the target still exists, and the action remains within cost and rate limits. Use an immutable action ID or content hash so changed arguments require a new approval.

This also reduces confused-deputy risk: untrusted page, email, or document content must never be able to turn an approval for one destination into a different tool call.

What should the implementation enforce?

• Maintain a server-side registry mapping each tool to allowed callers, data scopes, risk class, and approval rules.
• Give tools narrowly scoped credentials; do not give the model a general administrator token.
• Validate arguments against a strict schema and enforce business authorization independently of the model output.
• Queue pending actions with expiry, sanitized preview, approver identity, and an immutable argument digest.
• Record proposed, approved, rejected, executed, failed, and rolled-back states without logging secrets.
• Make retries idempotent so an approval cannot produce duplicate external actions.

NIST’s AI Risk Management Framework provides a broader Govern–Map–Measure–Manage structure. An approval gate belongs inside that system of ownership, risk tolerance, monitoring, and incident response; it is not a substitute for those controls.

How should the approval flow be tested?

• Try to call every high-risk tool without approval.
• Modify an argument after approval and verify execution is refused.
• Use an expired approval and a reviewer without permission.
• Insert a malicious instruction into retrieved content and confirm it cannot approve or redirect the action.
• Retry an approved request and verify the side effect occurs once.
• Test rejection, timeout, tool failure, audit logging, and rollback.