Benchmarks (Beta)

Two questions decide whether Aito fits a problem: is it accurate enough? and is it fast enough? This section answers both with reproducible numbers — accuracy against tuned ML baselines and public datasets, and throughput/latency against the kind of engine you'd otherwise reach for.

We try to be honest about where Aito wins and where it doesn't.

Where Aito competes — and where it doesn't

Aito is a predictive database: the same from/where you filter with is the evidence for predict / recommend / relate, with no training step and an explanation ($why) behind every result. That framing sets what the benchmarks should — and shouldn't — measure.

• Prediction quality is the point. Against tuned gradient-boosting and AutoML pipelines (Random Forest, LightGBM, FLAML) and against an LLM+RAG setup, Aito is strongest exactly on the hard, high-cardinality, cross-table targets a bag-of-features model struggles with — on the current baselines it tops the invoice-routing and expense suites, while purpose-built fine-tuned NLP models still lead on Banking77 — and it gets there with zero training time and full explainability. That's the differentiator; that's what these suites measure first.
• Raw query speed is not our competition area — and that's fine. For plain indexed search and filtering, a specialized engine (Elasticsearch, a native C++ columnar store) will be faster, and we don't try to beat them at their own game. Aito's bar on that axis is "not too slow": fast enough to serve interactive queries while carrying the prediction layer those engines don't have. The Performance page states this plainly, with numbers.

The trade you're evaluating is: a single system that predicts, recommends, relates and searches over your live data with no model-training pipeline — versus stitching a search engine to a separately-trained, separately-served ML stack. These benchmarks are here to show that trade is a good one on quality, and acceptable on speed.

Accuracy suites

Each suite is a real task with a held-out test set and published baselines, run as a server-side booktest so the numbers are reproducible. We're starting with a focused set and will extend it with more common ML benchmarks over time.

• Automated GL coding / invoice routing — predict processor, acceptor and GL code from invoice text with both the v1 and v2 engines, vs Random Forest / LightGBM / FLAML AutoML. On the current baseline (after the engine's scoring redesign) Aito v2 leads top-1 on all three targets with 0 s training — decisively on acceptor (v2 68.5% vs FLAML 59.5%), within the n=200 noise band on GL code and processor, so read those two as parity with a tuned AutoML search. FLAML remains the strongest baseline. The rank story backs it up: v2's link priors put the true acceptor at mean rank 1.8 (v1 2.2) and cut the processor rank ~3× vs v1.
• Expense categorization — predict an invoice's expense category (48 classes) from vendor, tax rate, amount and line text, vs Random Forest, LightGBM and FLAML AutoML (an LLM + RAG baseline is a pending slot). The headline: Aito v1 tops the top-1 table (79.0% vs LightGBM's 74.0%) — though the top is a tight cluster inside a test-set CI — and Aito is well-calibrated out of the box, with 0 s training and a $why behind every prediction, where the tree defaults are over- or under-confident. Reports accuracy, calibration (ECE / Brier / gap), and training time.
• Intent classification (Banking77) — 77-way fine-grained intent on 13k banking queries. 82.6% top-1 with zero training, shown in honest context against fine-tuned transformer sentence-encoders (which reach the low-90s and win on raw accuracy).

On the roadmap: text classification (Reuters, BBC News) and retrieval / RAG (BEIR SciFact, where Aito's inverted index feeds retrieval-augmented generation).

Performance

• Write & query performance — ingest throughput, query latency, and how they scale — with an honest read on how Aito compares to a specialized search engine, and where the prediction latency lands.

How the numbers are produced

Every figure comes from a run, not a slide. Accuracy suites are booktests over fixed test sets; performance figures come from the InvoicePerf / FlatInvoicePerf harnesses. The specific numbers on these pages — including the ones cited above — are generated at documentation-build time: a generator reads each run's committed metrics.json and interpolates it into the page, so they update themselves on re-baseline and a stale hand-typed digit can't slip through. The build fails loudly on any unresolved metric token.