Write & query performance (Beta)

The honest version first: for raw indexed search and filtering, a specialized engine like Elasticsearch (or a native C++ columnar store) will be faster than Aito, and we don't try to beat them at it. Aito's job on this axis is to be fast enough to serve interactive queries while carrying a prediction layer those engines don't have. This page states where Aito actually lands — including a direct Elasticsearch head-to-head.

What we measure

• Query / search latency — filter and full-text queries over live data.
• Predict / recommend / relate latency — the inference layer; there is no equivalent in a search engine, so it's an Aito-only number, not a comparison.
• Write / ingest — how fast new rows become queryable.
• Scaling — how each of the above moves as the table grows, on a single-vintage ladder from 1k to 10M rows (v1) and 1k to 1M (v2).

Query & search latency

Filter and full-text queries resolve out of the memory-mapped inverted index. Here is a direct head-to-head against Elasticsearch 8.15 on identical data and queries — the same 100k flat invoice rows loaded into Aito v1, Aito v2 and Elasticsearch, each queried through one shared HTTP client so client overhead is identical. Warm p50 latency (all three return the same hit counts):

Query	Aito v1	Aito v2	Elasticsearch
Exact filter (company = N)	10 ms	9 ms	3 ms
Full-text (description match)	7 ms	35 ms	3 ms
Filter + full-text	5 ms	6 ms	3 ms

_{Provenance: dataset invoice (100k flat rows) · warm p50 latency · Aito v1/v2 in-JVM vs Elasticsearch 8.15 in Docker on the same host · from booktest BookInvoiceEsBench es-head-to-head. Build-time tokens, not hand-typed.}

Read it straight: Elasticsearch is fastest on every query — a couple of milliseconds, roughly 2–3× quicker than Aito on filters. That is the expected result and exactly the one we promised — a specialized search engine beats us on raw indexed search. Aito still answers in single-digit to low-double-digit milliseconds — interactive — over the same index it uses for prediction. (Aito v2's full-text row is the outlier; the rep2 engine's full-text scan is a known beta slow spot, tracked separately — v1's full-text stays fast.)

The point isn't the milliseconds, it's that Aito stays interactive on the search axis and carries the calibrated-prediction layer a search engine doesn't have — see the comparison that matters. These are single-machine committed baselines (Aito in-JVM, ES in Docker on the same host); read the ratio, not the absolute milliseconds.

Prediction latency — v2 vs v1 (the part search engines don't have)

This is where the milliseconds buy something a search index can't do at all — a calibrated, explained prediction over your live data, no model to train or serve. Here is the v2 (rep2 / CollectionDb) engine measured head-to-head against the production v1 (TableDb) engine on the 10M-row linked-invoice benchmark (predicting processor / acceptor / glCode). Both columns are now the same engine vintage measured the same way, so accuracy is shown for both; the scaling curve behind it is the ladder. Latency is the mean per-predict time across the evaluation run (it includes the one cold first query — broken out separately below):

Predict target	v2 latency	v1 latency	v2 accuracy (end-to-end)	v1 accuracy (end-to-end)
acceptor	156 ms	146 ms	83.0%	71.2%
glCode	115 ms	305 ms	73.6%	78.7%
processor	691 ms	272 ms	70.8%	65.8%

_{Provenance: dataset invoice (InvoiceData synthetic, linked) · scale 10M rows · v2 from booktest InvoicePerf perf-optimized-10M-rep2, v1 from perf-optimized-10M · single dev machine, hardware unspecified. Every cell is a build-time metric token resolved from the committed perf snapshots — nothing here is hand-typed. Both engines: end-to-end top-1 over the same 511 held-out invoices, same corpus, same build, each at the heap its harness case sets (v1 12 GB, v2 4 GB).}

What this table says, and what it does not. v2 leads on both link targets — the ones that resolve a reference to a row in another table (processor, acceptor) — while v1 keeps glCode, the 32-class categorical. That split holds at every scale from 100k up in the ladder, so it is a property of the two engines rather than an artefact of one measurement. v1 remains several times faster per predict; that is the honest cost of v2's link-target accuracy, not a rounding difference.

An earlier version of this page withheld the v2 accuracy column, because the v2 10M snapshot was then a different engine vintage from the v1 column and computed over a different denominator — printing them side by side would have compared two measurements of two builds. Both halves of that objection are now gone: every cell above and in the ladder comes from one build, one corpus and the same end-to-end denominator.

What the latency columns do support: v2 is much faster on glCode (115 ms vs 305 ms), level on acceptor (156 ms vs 146 ms), and clearly slower on processor (691 ms vs 272 ms) — that high-cardinality cross-table link runs through rep2's layered cursor stack, which is the active latency-tuning target. Each predict ranks hundreds of candidates and returns a $why factor tree.

Warm-up — first query, cold

The first query per target pays JIT, file-cache miss and dictionary first-touch. v2's memory-mapped columnar reads make cold-start markedly faster on the two lower-cardinality targets (no big heap structures to materialize on first touch); on the high-cardinality processor target the two engines are roughly level — both pay ~10 s cold:

Predict target	v2 first query	v1 first query
acceptor	540 ms	2184 ms
glCode	317 ms	3407 ms
processor	9108 ms	9958 ms

_{Provenance: the first-query (0-1) row of the same InvoicePerf perf-optimized-10M{,-rep2} snapshots; build-time tokens, not hand-typed. Both columns are the same build as above.}

Memory & disk

v2 is built lean, which matters as much as speed when packing many tenants on a box:

• Memory-mapped, zero-allocation reads — a lookup reads straight out of the mapped column file, not a heap-materialized structure, so the resident footprint for a given dataset is lower than the row-oriented engine's.
• Compact on disk — a binary columnar format with adaptive byte widths, so the linkage and column layout stay smaller than a naïve fixed-width layout.
• Cheaper to build — rebuilding a table's optimized state streams through the columnar writer rather than materializing the whole table on the heap.

Scaling

What happens to accuracy and latency as the table grows? The InvoicePerf harness answers that on one corpus family — linked synthetic invoices — at five scales, predicting the same three targets each time: processor and acceptor (links into a ~46k-row employees table) and GL code (a 32-class categorical field).

Two properties make this ladder worth publishing, and neither held for the version of this table we shipped previously:

• One engine vintage. Every point below was measured on the same build, so the curve shows the effect of data scale — not the effect of an engine change that happened between two measurements taken months apart.
• 511 held-out test invoices at every point, on both engines, instead of 63. At n=63 the standard error on an accuracy near 50% is ~6pt, the same size as the effect the curve is meant to show; at n=511 it is ~2pt.

One correction comes with them. The accuracy published here is end-to-end: correct answers ÷ all test queries. A metric we reported earlier divided instead by only those queries whose correct answer appeared in the candidate list at all — a different quantity, with a denominator that moves per engine and per scale. Where candidate retrieval is below 100% it now gets its own column instead of being folded away.

Accuracy vs. rows — v1 (rep1 / TableDb)

Rows	Processor	Acceptor	GL code	Mean predict latency
1k	14.9%	17.2%	40.3%	36 ms
10k	15.5%	14.5%	66.9%	41 ms
100k	16.0%	27.0%	73.0%	45 ms
1M	49.1%	54.4%	74.4%	58 ms
10M	65.8%	71.2%	78.7%	172 ms

_{Provenance: dataset invoice (InvoiceData synthetic, linked) · v1 (rep1 / TableDb, /api/v1) · 511 held-out test invoices per point, each the rows immediately following that point's training slice · from booktest InvoicePerf perf-optimized-{K,10K,100K,M,10M} · single dev machine, hardware unspecified. End-to-end top-1 accuracy (v1 retrieves the correct candidate in 511/511 queries at every scale, so here end-to-end and retrieval-conditioned coincide). Latency is the mean per-predict time over the whole run and includes the one cold first query per target — broken out above. Every cell is a build-time metric token, not hand-typed.}

GL code climbs from the very first decade of rows (40.3% → 66.9% → 78.7%). With only 32 classes, even a 1k-row table puts real evidence behind each one, and every further decade keeps adding — with diminishing returns above 100k.

The two link targets are flat until 1M, then take off. Processor sits at 14.9% / 15.5% / 16.0% across 1k → 100k and only moves at 1M (49.1%) and 10M (65.8%). That is a property of the corpus, not a measurement artifact: there are ~46k employees, so a 100k-row table gives roughly two invoices per employee — far too thin to learn a routing signal for any individual. The link-target scaling story is really a 1M-and-above story, and we would rather say so than draw a smooth line through the flat part. (Acceptor's dip at 10k, 17.2% → 14.5%, is ~1.6 standard errors at n=511 — noise plus the fact that each scale is tested on a different held-out slice, not a regression.)

The same ladder on v2 (rep2 / CollectionDb)

Rows	Processor	Acceptor	GL code	Mean predict latency
1k	2.9%	18.4%	46.0%	82 ms
100k	33.7%	60.1%	72.8%	136 ms
1M	55.8%	74.4%	71.4%	159 ms
10M	70.8%	83.0%	73.6%	489 ms

_{Provenance: same corpus, same 511 held-out test invoices per point, same build · v2 (rep2 / CollectionDb, /api/v2) · from booktest InvoicePerf perf-{K,optimized-100K,optimized-M,optimized-10M}-rep2. End-to-end top-1 accuracy. The v2 ladder has no 10k point; 10M is now measured on this vintage, closing the last gap in the grid. Build-time tokens, not hand-typed.}

Read this ladder straight: from 100k up, v2 leads on both link targets and v1 leads on the categorical. At 1M, processor 49.1% (v1) vs 55.8% (v2) and acceptor 54.4% vs 74.4% go to v2, while GL code 74.4% vs 71.4% goes to v1 — and the same split holds at 100k and at 10M. v1 answers faster throughout (58 ms vs 159 ms at 1M).

This is a scaling ladder first: what it establishes is that both engines improve monotonically with rows. Read as a v1-vs-v2 verdict it is a split decision, not a win for either — and it is not the whole picture, because top-1 alone says nothing about how well-calibrated a prediction is.

_{A previous version of this page stated the opposite here — that v1 led v2 on all three targets at 1M. That sentence was written from metrics generated before the v2 scoring fix and never regenerated afterwards, so it described an engine that no longer existed while the table beside it rendered current numbers. The whole ladder is now re-measured on one build.}

Candidate retrieval — why v2 starts so low

Top-1 accuracy can only be as good as the candidate list it ranks. v1 offers the whole company-scoped employee set regardless of whether it has ever seen that employee; v2's candidate scan at small scale offers only employees it has actually observed in the training rows. That is the entire explanation for v2's 1k row above — and it is also a genuinely interesting scaling curve of its own, because retrieval converges to 100% as rows arrive:

Rows	Processor	Acceptor	GL code
1k	4.7%	34.1%	100.0%
100k	87.3%	99.6%	100.0%
1M	100.0%	100.0%	100.0%

_{Share of the 511 test queries in which the correct value appeared anywhere in the returned candidate list, from the same InvoicePerf -rep2 snapshots. v1's retrieval is 100.0% at every scale and on every target, so it has no such column.}

At 1k, v2 retrieves the correct processor in only 4.7% of queries; by 1M it is 100.0%. Dividing top-1 hits by retrieved queries instead of all of them is what made the old v2 small-scale numbers look strong — the denominator was ~5% of the test set. The columns above are the honest decomposition of that.

Mean rank falls too

Accuracy is a top-1 threshold, so a curve built on it can in principle move for threshold reasons alone. Mean rank of the correct answer is measured on a different footing and corroborates the same story independently:

Rows	Processor	Acceptor	GL code
1k	31.6	39.9	5.4
10k	25.9	65.4	2.4
100k	24.8	45.2	1.9
1M	9.9	18.8	1.5
10M	6.5	11.3	1.3

_{Mean 0-based rank of the correct value in the returned candidate list (lower is better) · v1 (rep1) · same InvoicePerf snapshots and same 511 test invoices. Rank is only defined when the correct value was retrieved at all; v1 retrieves it in every query at every scale, which is why this table is v1-only — a v2 rank column at 1k would be an average over ~24 queries and would read far better than the engine actually performed.}

Processor's correct answer moves from rank 31.6 to 6.5 out of a ~46k-employee candidate space, and GL code from 5.4 to 1.3 out of 32 — i.e. usually first or second. Note this improves through the 1k → 100k stretch where processor's top-1 accuracy is flat: the ranking is already getting better there, it just hasn't crossed the top-1 threshold yet.

Latency vs. rows — the sub-linear claim

This is the second story, and it holds cleanly now that it is measured on one vintage:

Rows	v1 mean predict latency	v2 mean predict latency
1k	36 ms	82 ms
10k	41 ms	—
100k	45 ms	136 ms
1M	58 ms	159 ms
10M	172 ms	489 ms

_{Mean per-predict latency across the processor, acceptor and GL-code targets, including the one cold first query per target · same InvoicePerf runs · single dev machine, hardware unspecified. Build-time tokens, not hand-typed.}

From 1 024 rows to 10 485 760 — four orders of magnitude — v1 predict latency goes from 36 ms to 172 ms. Four orders of magnitude of rows buy well under one order of magnitude of latency: the growth is roughly logarithmic in row count, not linear. v2 shows the same shape over the three decades it covers (82 ms → 159 ms).

The mechanism is that a prediction's cost is dominated by ranking candidates, not by scanning rows — the memory-mapped columnar layout resolves the evidence for a candidate in time that depends on the candidate set, not on table size. The candidate set here (~46k employees, 32 GL codes) is fixed by the schema, so adding rows adds evidence per candidate rather than work per query. Read the shape; the absolute milliseconds move with hardware.

What this ladder does not say

• No cross-method comparison at scale. The RF / LightGBM / FLAML baselines on the GL-coding page exist only at the 10k-row point. Nothing here supports "the tree models plateau while Aito keeps climbing" — we haven't run them at 1M or 10M. That comparison is pending, and until it exists, the ladder is a statement about Aito alone.
• The test slice differs per point. Each scale is evaluated on the 511 rows immediately after its own training data (the harness requires the test stream to start where training ended), so this is a proper held-out setup but not a single fixed test set shared across N.
• Synthetic corpus, single seed, single machine. Absolute values are point estimates; the direction is the durable signal.
• v2 at 10M is missing, as noted above. An older v2 10M measurement exists and is discussed in the 10M prediction table, but it predates both the scoring redesign and the end-to-end accuracy fix, so it is deliberately kept out of this ladder rather than mixed into it.

Every figure above is a committed baseline from a single dev machine, regenerated from the InvoicePerf harness snapshots at docs-build time (invoice-{table,collection}-* metrics), so it can't silently drift.

Writes

Writes are incremental: inserted rows are queryable and usable as prediction evidence immediately — there is no separate reindex or model-retraining step (contrast the accuracy suites, where every ML baseline needs a full refit to absorb new data). Batched ingest uses a back-pressured write path with deferred segment merging so read latency stays stable under load.

Measured v2 ingest on the linked-invoice benchmark (batched 2,000-row commits through /api/v2):

Metric	v2 ingest
Throughput	21,716 rows/s
Per-commit latency (p50)	90 ms
Per-commit latency (p90)	107 ms

_{Provenance: dataset invoice · batched 2,000-row commits through /api/v2, deferMerge default · from booktest BookInvoiceIngestBench ingest-invoices-v2 · single dev machine, hardware unspecified. Build-time tokens, not hand-typed.}

Deferred segment merging — the default write path — materially raises ingest throughput versus eager merging by batching the index rebuild instead of paying it on every commit. (The committed ingest snapshot measures only the deferred default; a tokenized eager-vs-deferred A/B is pending, so no fixed multiplier is quoted here.)

Like the scaling table, these are single-machine committed baselines generated from the BookInvoiceIngestBench snapshot — read the magnitude (tens of thousands of rows/s, sub-100 ms commits), not the exact figure. Write throughput is markedly more sensitive to machine load than query latency: on a contended box the same ingest measures an order of magnitude slower, so the baseline is captured on an unloaded run.

The comparison that matters

Against a specialized search engine, expect Aito to be competitive but not category-leading on raw query speed, and incomparable on prediction — because the search engine simply doesn't predict. The evaluation you're really making isn't "Aito vs Elasticsearch on QPS"; it's "one system that searches and predicts over live data with no training pipeline" vs "a fast search engine bolted to a separately-trained, separately-served ML stack." On the first axis you trade some raw speed; on the second you remove an entire moving part.

Related

• Benchmarks overview · GL-coding accuracy · Clusters & scaling · Warming strategies