Expense categorization (Beta)

Task: given a purchase invoice — its vendor, tax rate, amount, line count and free-text description — predict the expense category it should be booked to (48 categories). This is the everyday accounts-payable automation problem: every finance team codes invoices to a chart of accounts, and most of the work is repetitive and vendor-driven.

The realistic wrinkle is that categories are vendor-driven with genuine ambiguity: a given vendor books mostly to one category, sometimes a second, occasionally somewhere the amount or description implies — plus irreducible noise. A model that just memorizes "vendor → most common category" hits a ceiling (~58% here); doing better means modelling the ambiguity and reading the text, and saying how sure it is.

The data. A synthetic but realistically-structured corpus (benchmarks/gen_corpus.py): 10 000 training invoices, a 200-row held-out test set, 48 categories, a 6.6% base rate, and 15% label noise (so ~85% is the accuracy ceiling and calibration is meaningful). One corpus is consumed identically by every method — Aito reads the normalized tables and follows the vendor → industry link itself; the baselines get the same join denormalized into a flat feature table — so the comparison is fair by construction.

Methods. Aito with both engines (v1 / rep1 and v2 / rep2), a tuned scikit-learn Random Forest, LightGBM, FLAML AutoML, and gpt-5-mini with RAG (retrieval over invoice history) — all scored on the identical test set with identical metric definitions. Every method emits the same accuracy + calibration + speed schema; reproduction lives in benchmarks/.

Top-1 accuracy, top-3, and rank

Method	Top-1	Top-3	Mean rank of true class
Aito v1	53.5%	82.0%	5.8
Aito v2	57.5%	83.0%	5.4
Random Forest	56.5%	82.0%	4.9
LightGBM	47.0%	78.5%	5.6
FLAML AutoML	57.5%	84.5%	5.2
LLM (gpt-5-mini + RAG)	57.0%	79.0%	6.6

Vendor-majority lookup oracle ≈ 58% top-1; base rate ≈ 6.6%; noise ceiling ≈ 85%. (LLM is n=100; the others n=200.)

Read honestly:

• Top-1 is a near-tie at the top: FLAML AutoML and Aito v2 lead together (both 57.5%), Random Forest a hair behind (56.5%), Aito v1 at 53.5%, LightGBM trailing (47.0%). Everyone lands near the vendor-lookup oracle — this task rewards modelling the ambiguity on top of the vendor signal, which is where calibration (below) and training cost separate them. AutoML matches Aito's accuracy — after a 3-minute hyperparameter search per target, vs Aito's 0 s.
• Random Forest has the best mean rank (4.9 vs Aito's 5.4). Where the tree ensemble is unsure it still spreads plausible mass across nearby categories; it's a fair win to report.

Calibration: does the confidence mean anything?

Accuracy alone hides whether a model knows when it's unsure — which is what a finance team needs to decide "auto-post" vs "route to a human". We report the expected calibration error (ECE, lower is better), the top-1 Brier score (lower is better), and the calibration gap (mean confidence − accuracy; >0 means overconfident, <0 underconfident).

Method	ECE	Brier	Calibration gap
Aito v1	0.086	0.248	0.012
Aito v2	0.117	0.261	-0.042
Random Forest	0.157	0.277	-0.141
LightGBM	0.429	0.426	0.429
FLAML AutoML	0.124	0.267	-0.081
LLM (gpt-5-mini + RAG)	0.240	0.290	0.162

This is where the methods separate:

• Aito is well-calibrated out of the box. v1's confidence tracks its accuracy almost exactly (gap +0.01, ECE 0.079); v2 is very slightly underconfident (−0.04). No temperature-scaling or post-hoc calibration step — it falls out of the probabilistic model.
• Random Forest is underconfident (gap −0.14): its averaged-tree votes hedge, so it under-claims on the calls it gets right.
• LightGBM is badly overconfident (gap +0.43, ECE 0.43): its softmax outputs are sharply peaked, so it asserts ~90% confidence on predictions that are right far less often. On a task with real ambiguity, that is the dangerous failure mode.

Why it matters: AP automation runs on a confidence threshold — auto-post above X, route the rest to a human. A well-calibrated 57% model auto-posts the right slice safely; an overconfident model auto-posts its confident-but-wrong predictions straight into the ledger. Calibration, not raw top-1, is what makes the accuracy usable.

The number that isn't accuracy: training time

Method	Time to absorb new data (update the model)
Aito (v1 or v2)	0 s — incremental; new rows are queryable immediately
Random Forest	1 s (full refit)
LightGBM	168 s (full refit)
FLAML AutoML	192 s (budgeted search)

Every tree/AutoML baseline needs a retraining pipeline to absorb new invoices; Aito absorbs them on insert, with a $why explanation behind every prediction. On AP coding — where a new vendor or a re-mapped account shows up weekly — "0 s and explainable" is a large part of the real total cost.

Predict latency

The mirror image of training time. Aito does its learning at predict time (no model to pre-fit), so a single prediction costs tens of milliseconds; the tree models, once trained, infer in well under a millisecond.

Method	Predict (ms, mean)	p90 (ms)	Training
Aito v1	35.5	46.6	0 s
Aito v2	64.4	91.1	0 s
Random Forest	0.51	0.51	1 s
LightGBM	0.42	0.42	168 s
FLAML AutoML	0.14	0.14	192 s
LLM (gpt-5-mini + RAG)	14636	22212	0 s (no fit; builds a retrieval index)

So the honest read is: for a single field, a pre-trained tree serves faster — Aito's predict-time inference is tens of ms. What Aito buys with that is no training pipeline and any field predictable on demand — which is exactly what the next number is about.

Coding a whole invoice: multi-target latency (rep1 vs rep2)

Real AP coding isn't one prediction — a clerk codes several fields per invoice: the expense category, a cost center, an approver, a handler. A tree stack needs a separately-trained model per field; Aito predicts all four from the same data with no per-target training. This is where the per-invoice latency — and the rep1 vs rep2 engine cost — actually lands. approver/handler are the interesting ones: org-scoped, high-cardinality employee links, exactly the cross-table target rep2's link priors target.

Field	Kind	rep1 top-1	rep2 top-1	rep1 (ms)	rep2 (ms)
category	category link	53.5%	57.5%	20 ms	30 ms
cost_center	categorical	25.5%	30.0%	21 ms	27 ms
approver	employee link (org-scoped)	38.0%	58.0%	20 ms	53 ms
handler	employee link (org-scoped)	38.5%	33.5%	18 ms	50 ms
total / invoice	all four fields			79 ms	160 ms

The total is the per-invoice cost of coding all four fields on one engine, with 0 s training either way. rep1 and rep2 are close on this workload; rep2's link-prior machinery costs a little more on the org-scoped employee targets and buys better ranking of the true candidate (see the GL-coding benchmark for the rank story at depth).

Does it scale — and where does each method break?

Real AP data spans a huge range: a brand-new tenant has a handful of invoices; a large bureau has hundreds of thousands. A method is only useful if it works across that range. We swept train size 1 → 100,000 on a fixed 200-row hold-out (top-1 accuracy):

N (train)	1	10	100	1,000	10,000	100,000
Aito v1	—	—	7%	33%	56%	53%
Aito v2	—	—	8%	40%	57%	56%
Random Forest	5%	7%	10%	32%	57%	58%
LightGBM	✗	8%	4%	20%	52%	52%
FLAML AutoML	✗	✗	✗	35%	59%	59%
LLM (gpt-5-mini + RAG)	—	10%	23%	43%	50%	—

_{For the LLM, N is the history it retrieves from (RAG, k=30, n=30): more past invoices → better retrieval → higher accuracy. ✗ = couldn't train (too few samples / per class). — = not swept.}

There are three regimes, and the honest read is that no single method owns all of them:

• Tiny data (N ≤ 100): almost nothing works. Trees and Aito sit at the 6.6% base rate; LightGBM can't train at N=1 (needs ≥2 samples) and AutoML can't at N ≤ 100 (too few per class for cross-validation). The LLM+RAG has the same problem from the other side — with only 10–100 past invoices there's little relevant history to retrieve, so it's at 10–23%. Everyone is starved of signal here.
• Mid data (N = 1,000): Aito v2 and the LLM+RAG lead (40% and 43%) vs AutoML 35%, RF 32%, LightGBM 20%. Aito's priors and the LLM's retrieval both reach useful accuracy with less data than the tree/AutoML models need — the practical win for a tenant that isn't huge yet.
• Big data (N ≥ 10,000): everyone converges (~50–59%). AutoML and RF edge raw top-1; Aito matches within a couple of points; the LLM+RAG is right there too — all with 0 s training for Aito and the LLM, a growing training bill for the rest.

The LLM's whole curve is the RAG-needs-history story: 10% → 23% → 43% → 50% as the retrievable history grows 10 → 10k. It's an accurate model if you give it the past invoices to retrieve — but it pays for that parity in latency (≈15 s per prediction vs Aito's ~60 ms) and calibration (more overconfident), and it can't predict anything a plain retrieval wouldn't already surface.

Training cost is the other half of the story — it grows with data for the tree and AutoML models, and is zero for Aito and the LLM+RAG:

N (train)	1	10	100	1,000	10,000	100,000
Aito (v1/v2)	0 s	0 s	0 s	0 s	0 s	0 s
Random Forest	0 s	0 s	0 s	1 s	1 s	43 s
LightGBM	✗	0 s	5 s	42 s	177 s	215 s
FLAML AutoML	✗	✗	✗	62 s	60 s	62 s

So the shape of the whole benchmark: Aito is the only method usable across the entire range on every axis at once — never fails to "train", competitive-to- leading accuracy, best-calibrated, ~60 ms predictions, flat 0 s training. Each alternative gives one of those up: trees/AutoML pay a growing training bill and collapse at tiny N; the LLM+RAG matches the accuracy once it has history to retrieve, but at ~15 s/prediction and weaker calibration. Generated by benchmarks/scale_sweep.py + the per-scale eval booktest; numbers in benchmarks/results/scale-sweep.json.

How to read this

• Statistics: n=200 test (95% CI ≈ ±7pp at 50%), single synthetic seed. Read the direction — which method handles the ambiguity and stays calibrated — as the durable signal; treat individual cells as point estimates.
• The baselines are given the vendor→industry join for free, so this isolates modelling + calibration + training cost, not who saw which column.
• We report where Aito loses, not only where it wins.

Related

• Benchmarks overview · GL coding / invoice routing · Performance