Confidence-Gated Handoff

Calibrated confidence as governance: high $p auto-resolves, a borderline read goes to the LLM with Aito's shortlist, a low one goes to a human with the tentative answer attached, and sensitive intents are gated regardless. The number decides who acts.

Confidence-Gated Handoff: screenshot from the Agent demo
🤖 Agent_predictCross-vertical
In the demoIn the agent demo's live queue, 6 of 10 tickets auto-resolve, 1 goes to the LLM with Aito's shortlist, and the 3 that would move money or change a service (two refunds, a cancellation) wait for a human with the tentative read and its confidence attached. The policy: at or above 0.85 auto-resolve, 0.65 to 0.85 the LLM decides from Aito's shortlist, below 0.65 a human decides with the read attached, and refund or cancel gated regardless.

The problem

The hard question in automation is rarely whether the system can produce an answer. It is who is allowed to act on it. An LLM asserts with the same fluency whether it is right or guessing, so teams either gate everything behind review, which kills throughput, or trust everything, which is how the wrong refund goes out. Neither policy survives an audit conversation.

Governance needs a number, per decision, that says how sure the system is, and the number has to be calibrated: when it says 0.9, it should be right about nine times in ten. With that number, the automation boundary stops being a feeling and becomes a policy you can write down, defend, and tune.

How it works

_predict returns a calibrated $p with every prediction, and the $p drives the routing. In the demo the support queue is bucketed by the policy: at or above 0.85 the resolution auto-applies, between 0.65 and 0.85 the LLM decides from Aito's shortlist, and below 0.65 the item goes to a human with the tentative read attached, so triage starts from a draft instead of a blank queue. Anything that moves money, such as a refund or a cancellation, is gated for human action no matter how confident the prediction.

The thresholds are application policy, not model internals, so risk and compliance can set them per decision type and adjust them as trust accumulates. Every human correction writes back, which moves borderline cases toward the auto-resolve bucket over time. The same boundary works in front of an agent's actions; in the demo, the sales agent's drafted email always waits for a person to approve the send.

{
  "from": "resolutions",
  "where": {
    "text": "Hi, please refund the €45 charge on my roaming pack."
  },
  "predict": "intent",
  "limit": 3,
  "select": [
    "$p",
    "$value"
  ]
}

For the full architecture, see the technology overview. For the broader narrative across multiple use cases, read The Predictive Application.

See it live

This use case runs in the 🤖 Agent demo today. Click through to the live application and inspect the queries that produce the result.

Open the live demo →

Build it with Claude Code

Paste this prompt into Claude Code, or another coding agent, in your own codebase. It runs Aito locally from the Docker image, reads the API docs written for agents, and follows a working reference implementation.

Confidence-gated handoff: a prompt for Claude Code
Add confidence-gated handoff to this codebase using Aito, a predictive database. The feature: let our support agent resolve a ticket only when Aito's confidence clears a threshold, and hand the rest to a person.

1. Run Aito locally: `docker pull ghcr.io/aitohq/aito`, then follow https://aito.ai/docker/ to start it with your own API keys (keep them in the environment, never in code).
2. Read https://aito.ai/docs/api/llms.txt for the v2 query shapes before writing any query.
3. Read the use-case guide: https://aito.ai/use-cases/confidence-gated-handoff/
4. Use this reference implementation as the pattern (schema, data load and queries): https://github.com/AitoDotAI/aito-agent-demo/blob/main/src/gates.py and https://github.com/AitoDotAI/aito-agent-demo/blob/main/src/aito_client.py
5. Find the matching tables in this codebase, design the Aito schema for them (links between tables where we have foreign keys), and load a sample.
6. Measure before switching it on: hold out resolved cases and report the accuracy per confidence band, and the share the agent would handle at each threshold.
7. Let the agent act only above a threshold I choose; hand the rest to a person with the agent's draft and the reasons. Log every decision with its confidence.

Ask me before changing anything outside the new feature.

Frequently asked

Why does calibration matter more than accuracy here?

Because the policy acts on the number, not on the average. A 95%-accurate system whose confidence is uncalibrated still cannot tell you which 5% to send to a human. A calibrated $p sorts the individual cases, which is what an auto-versus-review boundary needs.

How do we choose the thresholds?

Start from the cost of a wrong auto-action per decision type. The demo's 0.85 and 0.65 are sensible defaults for support resolutions; an invoice approval might gate higher. Because $p is calibrated, the threshold maps directly to an expected error rate you can put in the policy document.

Does this work for gating LLM or agent actions, not just predictions?

Yes. The gate sits between any proposed action and its execution. In the demo's support queue the calibrated $p decides which tickets resolve on their own, and refunds and cancellations stay human-only regardless; the sales agent's drafted email always waits for a person to approve the send. The LLM proposes, the policy disposes.

What does the human see on a handoff?

The tentative read: the predicted intent, its $p, and the runner-up reads. Triage starts from a reviewable draft rather than a blank ticket, which is where the throughput gain on the human side comes from.

How does the boundary improve over time?

Corrections are rows. A human override enters the index immediately, so the next similar case predicts with better-grounded confidence, and categories that once sat in the review band drift above the auto threshold as evidence accumulates.