Benchmarks (Beta)

Two questions decide whether Aito fits a problem: is it accurate enough? and is it fast enough? This section answers both with reproducible numbers — accuracy against tuned ML baselines and public datasets, and throughput/latency against the kind of engine you'd otherwise reach for.

We try to be honest about where Aito wins and where it doesn't.

Where Aito competes — and where it doesn't

Aito is a predictive database: the same from/where you filter with is the evidence for predict / recommend / relate, with no training step and an explanation ($why) behind every result. That framing sets what the benchmarks should — and shouldn't — measure.

• Prediction quality is the point. Against tuned gradient-boosting and AutoML pipelines (Random Forest, LightGBM, FLAML) and against an LLM+RAG setup, Aito is competitive and often better at scale on the hard, high-cardinality, cross-table targets — the ones a bag-of-features model struggles with. And it gets there with zero training time and full explainability. That's the differentiator; that's what these suites measure first.
• Raw query speed is not our competition area — and that's fine. For plain indexed search and filtering, a specialized engine (Elasticsearch, a native C++ columnar store) will be faster, and we don't try to beat them at their own game. Aito's bar on that axis is "not too slow": fast enough to serve interactive queries while carrying the prediction layer those engines don't have. The Performance page states this plainly, with numbers.

The trade you're evaluating is: a single system that predicts, recommends, relates and searches over your live data with no model-training pipeline — versus stitching a search engine to a separately-trained, separately-served ML stack. These benchmarks are here to show that trade is a good one on quality, and acceptable on speed.

Accuracy suites

Each suite is a real task with a held-out test set and published baselines, run as a server-side booktest so the numbers are reproducible. We're starting with a focused set and will extend it with more common ML benchmarks over time.

• Automated GL coding / invoice routing — predict processor, acceptor and GL code from invoice text with both the v1 and v2 engines, vs Random Forest / LightGBM / FLAML AutoML. The headline: Aito wins on GL code and the hard cross-table processor target with 0 s training (and v2's link priors rank the true candidate far higher than v1).
• Intent classification (Banking77) — 77-way fine-grained intent on 13k banking queries. ~79% top-1 with zero training, shown in honest context against fine-tuned transformer sentence-encoders.
• Text classification (Reuters, BBC News) — coming soon.
• Retrieval / RAG (BEIR SciFact) — coming soon — where Aito's inverted index feeds retrieval-augmented generation.

Performance

• Write & query performance — ingest throughput, query latency, and how they scale — with an honest read on how Aito compares to a specialized search engine, and where the prediction latency lands.

How the numbers are produced

Every figure comes from a run, not a slide. Accuracy suites are booktests over fixed test sets; performance figures come from the InvoicePerf / FlatInvoicePerf harnesses. Some numbers on these pages are currently hand-transcribed from those runs and marked for generation at documentation-build time so they update themselves and can't drift — the mechanism that reads a run's metrics.json and interpolates it into these pages is being wired in.