Benchmarks (preliminary)

Two questions decide whether Aito fits a problem: is it accurate enough? and is it fast enough? This section answers both with reproducible numbers — accuracy against tuned ML baselines and public datasets, and throughput/latency against the kind of engine you'd otherwise reach for.

We try to be honest about where Aito wins and where it doesn't.

Results at a glance

Every figure below is generated from a committed benchmark run — follow any row to the page that earns it.

Accuracy, against the best tuned baseline on the same data:

What it costs to run, on a 10M-row linked-invoice database:

Where Aito competes — and where it doesn't

Aito is a predictive database: the same from/where you filter with is the evidence for predict / recommend / relate, with no training step and an explanation ($why) behind every result. That framing sets what the benchmarks should — and shouldn't — measure.

• Prediction quality is the point. Against tuned gradient-boosting and AutoML pipelines (Random Forest, LightGBM, FLAML) and against an LLM+RAG setup, Aito is strongest exactly on the hard, high-cardinality, cross-table targets a bag-of-features model struggles with — on the current baselines it tops the invoice-routing and expense suites, while purpose-built fine-tuned NLP models still lead on Banking77 — and it gets there with zero training time and full explainability. That's the differentiator; that's what these suites measure first.
• Raw query speed is not our competition area — and that's fine. For plain indexed search and filtering, a specialized engine (Elasticsearch, a native C++ columnar store) will be faster, and we don't try to beat them at their own game. Aito's bar on that axis is "not too slow": fast enough to serve interactive queries while carrying the prediction layer those engines don't have. The speed page states this plainly, with numbers.

The trade you're evaluating is: a single system that predicts, recommends, relates and searches over your live data with no model-training pipeline — versus stitching a search engine to a separately-trained, separately-served ML stack. These benchmarks are here to show that trade is a good one on quality, and acceptable on speed.

Accuracy suites

Each suite is a real task with a held-out test set and published baselines, run as a booktest so the numbers are reproducible. We're starting with a focused set and will extend it with more common ML benchmarks over time.

• Automated GL coding / invoice routing — predict processor, acceptor and GL code from invoice text with both the v1 and v2 engines, vs Random Forest / LightGBM / FLAML AutoML. On the current baseline (after the engine's scoring redesign) Aito v2 leads top-1 on all three targets with 0 s training — decisively on acceptor (v2 61.4% vs FLAML 56.2%) and GL code, and on processor (23.9% vs 19.4%) — on a 2,000-row hold-out with 95% intervals published per cell. FLAML remains the strongest baseline. The rank story backs it up: v2's link priors put the true acceptor at mean rank 1.0 (v1 2.1) and cut the processor rank ~5× vs v1.

• Expense categorization — predict an invoice's expense category (48 classes) from vendor, tax rate, amount and line text, vs Random Forest, LightGBM and FLAML AutoML (an LLM + RAG baseline is a pending slot). The headline: Aito v1 and v2 tie at the top of the top-1 table (79.8% each vs LightGBM's 76.1%) — a lead a paired test separates — and Aito is well-calibrated out of the box, with 0 s training and a $why behind every prediction, where the tree defaults are over- or under-confident. Reports accuracy, calibration (ECE / Brier / gap), and training time.

• Invoice line → product matching — match an incoming invoice line to a SKU in a live catalogue, as a cross-table link prediction, vs BM25 in three strengths. Against a catalogue-only lexical index — what most matching systems ship — 83.6% vs 64.4%; against BM25 given the same billing history — the honest comparison, and the one the page leads with — BM25 is ahead (89.2% vs 83.6%). That row measures the demo's own query, which names the supplier but not the product NAME the line is matching against; over the full sweep a query that names it reaches 91.5%, above the baseline. The page explains the difference. The differentiator is that the history layer is the database — corrections are queryable on insert, with no reindex. Also states the open defect (popular SKUs beating thin correct ones) and carries a partner-facing implementation section.

• Support ticket triage with an agent: five decisions per ticket (product, category, priority, resolution, first step), scored on held-out tickets that were never loaded, against an LLM alone, LLM + RAG and two hybrids. On tickets with new wording, Aito alone (61.9%) and LLM + RAG (63.4%) are level; the LLM alone reaches 26.9%. Aito's lead over all held-out tickets comes from wording it had already seen, and the page says so.

• Intent classification (Banking77) — 77-way fine-grained intent on 13k banking queries. 82.6% top-1 with zero training, shown in honest context against fine-tuned transformer sentence-encoders (which reach the low-90s and win on raw accuracy).

• Smart search & recommendations — rank the products a shopper was shown so the one they bought comes first, from the query, their profile and the products' attributes, on a generated shop log with a known answer key. recommend with basedOn (0.534 nDCG@10) beats BM25 given the same purchase history (0.468), mostly on wordings the catalogue does not contain; text search still wins on brand queries. Also the For You shelf, with no query.

On the roadmap: text classification (Reuters, BBC News) and retrieval / RAG (BEIR SciFact, where Aito's inverted index feeds retrieval-augmented generation).

Speed & footprint

• Query, predict & write speed — query latency against Elasticsearch on identical data, the v1-vs-v2 prediction head-to-head at 10M rows, and ingest throughput — with an honest read on how Aito compares to a specialized search engine.
• Memory & disk footprint — what the same 10M-row database costs to hold: 3,258 MB on disk, 2.3× smaller than the v1 engine, and why the heap doesn't grow with the corpus.
• Queries under writes — what a prediction costs on a database that is being written to the whole time the queries run: 853 ms at p50 under a continuous write load, and every answer byte-identical to a cold recompute.

How it scales

Two different questions get asked with the same word — how much data do I need? and how big can my database get? — and one page now answers both on one corpus (expense categorization) with one fixed hold-out, so a number on either axis is about the same data.

• Training size, 1 → 1 000 000 rows on a corpus that grows by adding tenants, every method on the same rows: Aito leads at N=10, 1 000 and 10 000 with no training step, and is tied at N=100 (at N=1 000, v2 is 64% against a field at 57%); at 100 000 LightGBM leads by about three points; at 1 000 000 LightGBM leads Aito v1 by about 2.4 points after a 2332 s training run, with Aito v1 about 1.6 points behind Random Forest. Published as the crossover it is.
• Database size, to 10 000 000 rows: predict latency, the cost of coding a whole invoice in one batched request, and heap / mapped / disk for both engines, each measured in a dedicated server process on the compacted state — v2 at 10M holds the database in 215 MB of heap against v1's 2,426 MB, both inside a 4 GB heap.
• The finer engine-to-engine curve on the linked invoice corpus, 1k → 10M, is the capacity ladder.

How the numbers are produced

Every figure comes from a run, not a slide. Accuracy suites are booktests over fixed test sets; performance figures come from the InvoicePerf / FlatInvoicePerf harnesses. The specific numbers on these pages — including the ones cited above — are generated at documentation-build time: a generator reads each run's committed metrics.json and interpolates it into the page, so they update themselves on re-baseline and a stale hand-typed digit can't slip through. The build fails loudly on any unresolved metric token.