How it scales (Beta)
Two different questions get asked with the word scaling, and they vary different things:
- How much data do I need? — accuracy as the training set grows. This decides whether you can start before you have a dataset.
- How big can my database get? — latency and memory as the corpus grows. This decides what a tenant costs to run at ten million rows.
This page answers both on the expense-categorization invoices, with one fixed 2,000-row hold-out scored at every point. The first axis runs every method against the same rows: Aito with both engines, Random Forest, LightGBM and FLAML AutoML. The second is Aito against itself, both engines, because that is the question it answers.
The training-size corpus grows the way a SaaS grows — by adding tenants, not by giving one tenant a longer memory. At every size it holds one organization per ~250 invoices, tenants of Zipf-distributed sizes, a standard chart of accounts shared across them, and each tenant's own tail of specialist vendors on top of the shared generalists. Ten times the rows is ten times the tenants.
Every figure is a build-time token resolved from benchmarks/results/ —
nothing here is hand-typed.
Provenance. benchmarks/scale_sweep.py generates a deterministic corpus
per scale and hands the same one to every method, Aito included. The
hold-out is pinned at a fixed offset and excluded from every training set,
so a 1M corpus scores the same 2,000 invoices as a 1k one and never trains on
them. Corpora at or above 1M are cached on disk and verified against their
manifest before reuse.
How much data do I need?
Real AP data spans a huge range: a brand-new tenant has a handful of invoices, a large bureau has millions. A method is only useful if it works across that range — and the honest finding is that no single method wins across it.
Top-1 accuracy by training-set size, every method on the same corpus and the same 2,000 held-out rows at each scale. At 1 000 rows the corpus has four organizations; at a million, four thousand — each with a few hundred invoices, its own specialist vendors, and the shared chart of accounts:
| Method | N=1 | N=10 | N=100 | N=1 000 | N=10 000 | N=100 000 | N=1 000 000 |
|---|---|---|---|---|---|---|---|
| Aito v2 | 2% | 10% | 28% | 64% | 79% | 83% | 84% |
| Aito v1 | 2% | 10% | 30% | 60% | 79% | 82% | 85% |
| Random Forest | 2% | 6% | 30% | 57% | 73% | 85% | 87% |
| LightGBM | ✗ | 5% | 15% | 56% | 77% | 86% | 88% |
| FLAML AutoML (60 s) | ✗ | ✗ | ✗ | 56% | 75% | 83% | 42% |
How to read. ✗ = the method could not run at that size, not a low score: LightGBM needs ≥2 samples (N=1); FLAML needs ≥2 classes, which a 1/10/100-row training set does not reliably contain. FLAML gets a fixed 60 s budget at every size; read its curve as "a fixed budget does not scale". Bold marks the best method in a column. The corpus caps accuracy at 90% by construction (10% label noise).
This is a crossover, and we publish it as one. Every rung is scored on the same held-out rows, so which method is ahead is settled by the rows where two methods DISAGREE, not by two overlapping intervals — and on this hold-out all four rungs are settled.
| Training rows | Best Aito | Best baseline | Difference | Rows only Aito | Rows only baseline | |
|---|---|---|---|---|---|---|
| 1 000 | v2 63.5% | RF 56.8% | +6.8 pt | 349 | 213 | p < 0.0001 |
| 10 000 | v1 79.2% | LightGBM 76.6% | +2.6 pt | 175 | 123 | p = 0.0031 |
| 100 000 | v2 82.8% | LightGBM 85.9% | -3.0 pt | 39 | 100 | p < 0.0001 |
| 1 000 000 | v1 85.3% | LightGBM 87.7% | -2.4 pt | 15 | 63 | p < 0.0001 |
Aito wins the small end and loses the large end, and both halves are real. With no training step it is ahead by +6.8 points at a thousand rows and +2.6 at ten thousand. Past that the boosted trees pass it: -3.0 points at a hundred thousand and -2.4 at a million. The crossover is between 10 000 and 100 000 rows. Earlier revisions of this page reported the same shape from a 200-row hold-out, where none of these four differences were distinguishable from chance; they are now, in both directions.
Why the crossover happens. This corpus identifies each of 1 200 invoice types by a combination of description tokens drawn from a vocabulary of only 238. Aito scores evidence per token and per learned conjunction, and it calibrates each piece from the data it has — which is exactly why it wins when there are a handful of examples per type: independent evidence is the best estimate you can make from that little. With hundreds of examples per type a tree model can learn which tokens co-occur, and the remaining ties — the true class and a near-neighbour sharing the same strong combination, decided by one leftover token — go the tree's way. We tested this rather than assumed it: a naive-Bayes classifier on the identical evidence reproduces Aito's curve to within a point at every scale, and giving it token pairs is what makes it climb again at a million rows. The ceiling is the combining rule, and it is the same property that makes the low-data lead.
What each side pays for its accuracy on this curve:
| At N=1 000 000 | Training run | Predict latency |
|---|---|---|
| Random Forest | 775 s | 0.2 ms |
| LightGBM | 2332 s | 0.3 ms |
| Aito v1 | 0 s | 90 ms |
| Aito v2 | 0 s | 151 ms |
Latency on this table is from the sweep, on a shared machine, single predicts, 2,000 rows. The capacity section below is the certified measurement; read these as ratios.
The two engines cross over too: v2 is stronger below ~1 000 rows; from 10 000 rows they are within a point or two of each other, with v1 ahead at a million.
Not comparable to the single-scale table on the expense page. The sweep scores one fixed hold-out at every scale; that is a different 2,000-row sample from the one the single-scale comparison uses, so the N=10 000 column here will not match it exactly. Within the sweep every cell is comparable; across the two tables, read the shape.
How big can my database get?
The second axis holds the task fixed and grows the corpus to ten million rows. It is measured on a single-tenant corpus — forty organizations at every size, so ten million rows is one very large customer rather than forty thousand small ones — because what it measures is what the database costs to hold and query, not how the evidence is distributed. The questions are what a prediction costs and what the database costs to hold — accuracy is reported because we have it, but on this axis it is the minor figure: the same invoices are being predicted against more history.
Measured on the DEPLOYED configuration. These rows are an optimized
(compacted) database, which is what a deployment runs. The same measurement
before compaction is materially worse — at 10 000 000 rows Aito v1 reads
1 638 ms mean and 3 790 ms p90 as-ingested against 415 and 580 optimized — so
an uncompacted figure would understate the product by four to six times. The
invoice ladder on the speed page is measured the
same way, so the two pages are comparable.
Accuracy on this axis is scored on 200 held-out rows (a ±5 point interval); it is reported because we have it, and the ladder above is where accuracy is measured properly.
| Aito v2, rows in the database | Warm predict, median | Warm p90 | Mean incl. cold | First query | …whole invoice batched | Top-1 |
|---|---|---|---|---|---|---|
| 1 000 000 | 62 ms | 116 ms | 88 ms | 120 ms | 125 ms | 82.5% |
| 10 000 000 | 279 ms | 823 ms | 601 ms | 160 ms | 369 ms | 82.0% |
| Aito v1, rows in the database | Warm predict, median | Warm p90 | Mean incl. cold | First query | …whole invoice batched | Top-1 |
|---|---|---|---|---|---|---|
| 1 000 000 | 52 ms | 95 ms | 107 ms | 101 ms | 97 ms | 82.0% |
| 10 000 000 | 210 ms | 417 ms | 355 ms | 354 ms | 429 ms | 83.0% |
How to read. Warm is the second half of the queries, once caches and
the JIT have settled — what a live server does all day, and the column to size
a deployment from. Mean incl. cold averages that together with the first
few requests after a restart, so it is pessimistic about steady state; both
are published so the ramp is visible rather than hidden, and first query
is the cost of that ramp. "Whole invoice" batched asks for all four fields —
category, cost centre, approver, handler — in one /_batch request, which is
what a real client does and what it should budget for; it is the minimum
over timed rounds on a shared machine, because contention only ever adds
time. Figures carry ~8% run-to-run spread; read ratios, not milliseconds.
What it costs to hold
Memory and disk for the same databases, read from the server's own /status
endpoints — the same source the invoice footprint
comes from, so the two are comparable by construction:
| Rows | Engine | JVM heap after GC | Live heap (reachable bytes) | Memory-mapped | On disk |
|---|---|---|---|---|---|
| 1 000 000 | Aito v2 | 578 MB | 549 MB | 578 MB | 284 MB |
| 1 000 000 | Aito v1 | 961 MB | 831 MB | 1,540 MB | 770 MB |
| 10 000 000 | Aito v2 | 1,720 MB | 1,560 MB | 5,766 MB | 2,829 MB |
| 10 000 000 | Aito v1 | 2,603 MB | 2,423 MB | 15,506 MB | 7,761 MB |
Two things stand out at 10M: the v2 engine holds the same database in substantially less heap than v1 and on a third of the disk, and it keeps far less of it on the heap at all — the mapped column is the data it serves from disk rather than resident memory. The latency trade that used to accompany that is now small: on the deployed configuration v1 answers a single predict in 355 ms against v2's 601 ms. Both are measured; neither is a target.
Why v1's mapped figure exceeds its on-disk size at 10M. optimize
rewrites segments, and the superseded ones stay memory-mapped until they are
released, so the mapped column double-counts for a while after compaction.
Mapped bytes are virtual address space backed by files, not resident memory —
the heap columns are what the process actually holds, and the on-disk figure
is the one to size storage from.
How to read. "JVM heap after GC" is System.gc() then used heap — the
figure the invoice ladder publishes, kept here so the two are like for like;
under a concurrent collector it can overstate. "Live heap" is jmap -histo:live, the sum of reachable bytes, which uncollected garbage cannot
inflate; when the two diverge, trust the second. Memory-mapped is the data the
engine serves from disk without holding it on the heap — the reason the heap
does not grow with the corpus.
The engine-to-engine ladder
The finer-grained curve — five corpus sizes from 1k to 10M on the linked invoice-routing corpus, both engines, with candidate retrieval and mean rank — is the capacity ladder. It is the evidence behind the 10M head-to-head on the speed page; it compares the engines to each other and carries no external baseline, which is why the expense corpus above leads.
What this page does not say
- Synthetic corpus, single seed, single machine. Every cell is one binomial proportion on 2,000 held-out rows, which puts a 95% interval of about ±2 points on it. Where we say one method is ahead of another, that comes from an exact McNemar test on the rows where the two DISAGREE, not from comparing two means; where the test does not separate them, the page says they are tied. This table was previously measured on 200 rows, where the interval was ±5 points and none of its differences were resolved.
- FLAML's column is a draw, not a fixed model. It searches for a wall-clock
budget, so a loaded machine explores fewer configurations and returns a
different model; the seed does not pin this. Each of its cells is the MEDIAN of
3 fits and
benchmarks/results/records every fit's top-1. The spread is small where the budget is comfortable (73.6–75.0% at N=10 000) and enormous where it is not: at N=1 000 000 the three fits came out 4.9–83.0%. That cell is not a measurement of AutoML so much as a demonstration that 60 seconds does not buy a model at a million rows — which is the point the column is here to make, and the reason we publish its range next to its median. - The corpus shares one chart of accounts across tenants. We also built a variant where a quarter of the codes are tenant-specific; there every method that combines evidence independently — Aito, Random Forest, naive Bayes — loses the relabelled rows and only boosted trees recover them. That is a property of the combining rule, recorded in the engine's tickets, and not a curve we publish as a benchmark.
- The baselines were not run at 10M. Training Random Forest at 1M already costs 775 s; the capacity axis is about what Aito costs to run, not a tenth comparison.
- Latency and memory are from a shared development machine, not a provisioned host; the numbers certify the shape, not a service level.