The problem
Operational ML systems need an operational dashboard. "Is the prediction working today, on the data flowing right now, at the confidence tiers we're auto-processing?" If the dashboard says yes, the team trusts the automation. If it says no — or if there's no dashboard at all — the team either over-trusts and produces silent errors, or under-trusts and misses the automation gains.
The dashboard isn't optional in a confidence-tiered automation system. It's the safety mechanism that makes the tiering credible. "Cost center: 96.0% right on 200 held-out purchases, against 34% for always guessing the most common value, and right as often as it claims in the top band" — that's the answer the dashboard needs to give for every field the system predicts on.
How it works
The quality dashboard runs _evaluate on a held-out sample of the live data: each row's field is hidden, predicted from the rest, and compared with the truth. Per field, it surfaces accuracy, the most-common-value baseline, and the gain over it, green when positive and red when not. Per confidence band (≥ 0.85, 0.5–0.85, < 0.5) it shows how often the top prediction was right next to what $p claimed, with the case count, and marks a band with too few cases to judge. In production the same query runs on a regular cadence (daily, hourly for high-volume tenants).
The dashboard is the operational view; calibration monitoring is the over-time view. The dashboard answers "how is the prediction performing right now?"; calibration monitoring tracks how that answer drifts week-over-week. Different audiences read each: the AP lead reads the dashboard daily, the data team reads the drift tracker weekly.
{
"testSource": {
"from": "purchases",
"limit": 200
},
"evaluate": {
"from": "purchases",
"where": {
"supplier": {
"$get": "supplier"
},
"description": {
"$get": "description"
},
"amount_eur": {
"$get": "amount_eur"
}
},
"predict": "cost_center"
},
"select": [
"accuracy",
"baseAccuracy",
"cases"
]
}For the full architecture, see the technology overview. For the broader narrative across multiple use cases, read The Predictive Application.
See it live
This use case runs in 2 live demos today, the 📋 ERP and 🧾 Accounting demos. Each link opens the view that shows it, where you can inspect the queries that produce the result.
Frequently asked
How often should the dashboard refresh?
Daily is the floor for non-trivial volume; hourly for high-volume tenants where the auto-process tier sees large transaction counts. Less frequent refreshes mean drift gets noticed late — by the time you see the dashboard, the bad predictions are already in the books.
What confidence buckets should we use?
The ERP demo uses three: ≥ 0.85, the auto-approve zone, then 0.5–0.85 and < 0.5, and greys out a band with too few cases for its accuracy to mean anything. The top band is the one to check most often. Finer bands need more held-out cases in each; choose them from your own volume.
Can the dashboard surface per-tenant calibration in multi-tenant deployments?
Yes. Run _evaluate with a customer_id filter; the calibration is per-tenant. The accounting.aito.ai demo runs this pattern for each of its 255 tenants — each tenant's evaluation reflects its own data.
What action should we take when a field's calibration breaks?
Lower the auto-process threshold for that field, so manual review absorbs the uncertainty. If the top band's accuracy falls well below what $p claims there, pause auto-processing on that field until the underlying cause is identified.
Is the dashboard built into Aito or do we build it ourselves?
Aito returns the metrics; the dashboard UI is yours to build (or reuse). The accounting demo includes a reference implementation of the quality dashboard you can fork. Most deployments customise the dashboard to fit their existing operational tooling.
