Expense categorization (Beta)

Task: given a purchase invoice — its vendor, tax rate, amount, line count and free-text line description — predict the expense category (GL account) it should be booked to (48 categories). This is the everyday accounts-payable automation problem: every finance team codes invoices to a chart of accounts, and most of the work is repetitive.

The realistic wrinkle is that the account is determined by what was bought, not who sent it. Big generalist vendors sell across many accounts, so a model that just memorizes "vendor → its most common category" barely clears the ~20% vendor-lookup oracle. Doing better means reading the line-item text — which identifies the invoice type, and the type determines the account — modelling the residual ambiguity, and, on top of irreducible label noise, saying how sure it is.

The data. A synthetic but realistically-structured corpus (benchmarks/gen_corpus.py, seed 41): 10 000 training invoices, a 200-row held-out test set, 48 categories, a ~3% base rate (most-common class), and 10% label noise (so ~90% is the accuracy ceiling and calibration is meaningful). Categories are emitted from a latent invoice-type topic model: each type sprays a bag of description keywords, so the signal accumulates across the text rather than sitting in any single feature. One corpus is consumed identically by every method — Aito reads the normalized tables and follows the vendor → industry link itself; the baselines get the same join denormalized into a flat feature table — so the comparison is fair by construction.

Methods. Aito with both engines (v1 / rep1 and v2 / rep2), a tuned scikit-learn Random Forest, LightGBM, and FLAML AutoML — all scored on the identical test set with identical metric definitions. Every method emits the same accuracy + calibration + speed schema; reproduction lives in benchmarks/. An LLM + RAG baseline (gpt-5-mini, retrieval over invoice history) is a slot that still needs re-running on this corpus — see below.

Top-1 accuracy, top-3, and rank

Method	Top-1	Top-3	Mean rank of true class
Aito v1	79.0%	90.5%	3.1
Aito v2	75.0%	88.5%	3.6
Random Forest	69.5%	91.0%	3.1
LightGBM	74.0%	88.5%	3.7
FLAML AutoML	71.0%	89.0%	3.4
LLM (gpt-5-mini + RAG)	—	—	—

Vendor-lookup oracle ≈ 20% top-1; base rate ≈ 3%; 10% label noise caps accuracy near 90%. n=200 test. (The LLM row is a pending slot — see below.)

_{Provenance: dataset expense-categorization (benchmarks/gen_corpus.py, seed 41) · scale 10 000 train / 200 test, 48 categories · Aito from booktest ExpenseCategorizationEvaluation (v1 = rep1, v2 = rep2) · RF / LightGBM / FLAML AutoML from the Python baselines in benchmarks/baselines. Every result figure is a build-time metric token resolved from a committed metrics.json — nothing in the tables is hand-typed.}

Read honestly — the top is a tight cluster:

• Aito v1 leads raw top-1 (79.0%), with Aito v2 (75.0%), LightGBM (74.0%), FLAML AutoML (71.0%) and Random Forest (69.5%) in a close pack behind it. (Earlier baselines of this page had LightGBM ahead; the engine's scoring redesign moved both Aito columns.) Every method lands 3–4× above the ~20% vendor oracle — they are all reading the line-item text, which is where the account signal lives. The spread across the top is a few points, inside a couple of test-set CIs (±7pp), so top-1 alone doesn't separate them — calibration and training cost do.
• Random Forest and Aito v1 are effectively tied for the best mean rank (both ≈3.1, RF a hair ahead), and RF keeps the best top-3 (91.0% vs Aito v1's 90.5%): where the ensemble is unsure it still spreads plausible mass across nearby categories. Aito v2's mean rank (3.6) trails the pack — an honest v2 soft spot on this corpus.
• LightGBM stays close on top-1 but with confidence it can't back up — that is the calibration story below.

Calibration: does the confidence mean anything?

Accuracy alone hides whether a model knows when it's unsure — which is what a finance team needs to decide "auto-post" vs "route to a human". We report the expected calibration error (ECE, lower is better), the top-1 Brier score (lower is better), and the calibration gap (mean confidence − accuracy; >0 means overconfident, <0 underconfident).

Method	ECE	Brier	Calibration gap
Aito v1	0.098	0.159	0.045
Aito v2	0.086	0.169	0.021
Random Forest	0.286	0.273	-0.285
LightGBM	0.185	0.207	0.170
FLAML AutoML	0.077	0.182	0.042
LLM (gpt-5-mini + RAG)	—	—	—

This is where the methods separate:

• Aito is well-calibrated straight out of the model — no temperature scaling or post-hoc step. v2 has the smallest calibration gap of any method (gap 0.021, ECE 0.086); v1 runs mildly hot (gap 0.045, ECE 0.098) while leading top-1.
• FLAML AutoML has the lowest ECE ( 0.077, gap 0.042), with Aito v2 right behind — but it only gets there after a budgeted per-target hyperparameter search (see training time); the tree defaults don't.
• Random Forest is badly underconfident (gap -0.285, the highest ECE at 0.286): its averaged-tree votes hedge, so it under-claims on the calls it gets right.
• LightGBM is overconfident (gap 0.170, ECE 0.185): its softmax outputs are sharply peaked — mean confidence 91.0% — so it asserts high confidence on predictions that are right less often. On a task with real ambiguity, that is the dangerous failure mode.

Why it matters: AP automation runs on a confidence threshold — auto-post above X, route the rest to a human. A well-calibrated model auto-posts its high-confidence slice safely; an overconfident model auto-posts its confident-but-wrong predictions straight into the ledger. Calibration, not raw top-1, is what makes the accuracy usable — and it is where Aito (with no training step) and AutoML (with a long one) pull ahead of the tree defaults.

The number that isn't accuracy: training time

Method	Time to absorb new data (update the model)
Aito (v1 or v2)	0 s — incremental; new rows are queryable immediately
Random Forest	2 s (full refit)
LightGBM	192 s (full refit)
FLAML AutoML	142 s (budgeted search)

Every tree/AutoML baseline needs a retraining pipeline to absorb new invoices; Aito absorbs them on insert, with a $why explanation behind every prediction. On AP coding — where a new vendor or a re-mapped account shows up weekly — "0 s and explainable" is a large part of the real total cost. Note the best-calibrated baseline, AutoML, is also the most expensive to keep current.

Predict latency

The mirror image of training time. Aito does its learning at predict time (no model to pre-fit), so a single prediction costs tens of milliseconds; the tree models, once trained, infer in well under a millisecond.

Method	Predict (ms, mean)	p90 (ms)	Training
Aito v1	51.1	67.8	0 s
Aito v2	87.3	138.1	0 s
Random Forest	0.48	0.48	2 s
LightGBM	0.70	0.70	192 s
FLAML AutoML	0.03	0.03	142 s
LLM (gpt-5-mini + RAG)	—	—	—

So the honest read is: for a single field, a pre-trained tree serves faster — Aito's predict-time inference is tens of ms. What Aito buys with that is no training pipeline and any field predictable on demand — which is exactly what the next number is about.

Coding a whole invoice: multi-target latency (rep1 vs rep2)

Real AP coding isn't one prediction — a clerk codes several fields per invoice: the expense category, a cost center, an approver, a handler. A tree stack needs a separately-trained model per field; Aito predicts all four from the same data with no per-target training. This is where the per-invoice latency — and the rep1 vs rep2 engine cost — actually lands. approver/handler are the interesting ones: org-scoped, high-cardinality employee links, exactly the cross-table target rep2's link priors target.

Field	Kind	rep1 top-1	rep2 top-1	rep1 (ms)	rep2 (ms)
category	category link	79.0%	69.5%	35 ms	43 ms
cost_center	categorical	51.5%	42.0%	30 ms	39 ms
approver	employee link (org-scoped)	57.0%	62.0%	34 ms	57 ms
handler	employee link (org-scoped)	66.0%	68.0%	29 ms	56 ms
total / invoice	all four fields			127 ms	194 ms

The total is the per-invoice cost of coding all four fields on one engine, with 0 s training either way. On the org-scoped employee links, rep2's link-prior machinery earns its extra latency: it lifts approver to 62.0% (rep2) from 57.0% (rep1), and ranks the true candidate better — see the GL-coding benchmark for the rank story at depth.

Does it scale?

Real AP data spans a huge range: a brand-new tenant has a handful of invoices; a large bureau has hundreds of thousands, and a method is only useful if it works across that range. Aito's real edge here is data-efficiency — reaching useful accuracy with less data than the tree/AutoML models need — plus flat 0 s training at every size.

LLM baseline — pending on this corpus

The realistic LLM deployment for this task is retrieval-augmented: for each invoice, retrieve the most similar historical invoices (same vendor, similar text) and put them in the prompt, so gpt-5-mini copies the account from the vendor's retrieved history rather than reasoning about opaque category codes. A raw few-shot prompt (no retrieval) has nothing to reason about and scores near the floor; RAG is what makes an LLM competitive here.

A prior RAG run reached parity with the tree/Aito cluster on the previous corpus vintage, but it has not been re-run on the current line-item corpus (it needs Azure credentials and costs ~15 s per prediction), so the LLM column reads — above. The archived prior result is kept at benchmarks/results/stale-oldcorpus/llm.json; benchmarks/baselines/run_llm.py --retrieval rag regenerates it against this corpus.

How to read this

• Aito v1 tops the top-1 table (79.0%), with v2, LightGBM, AutoML and RF a close pack behind — but the margin is inside a test-set CI, so the durable read is "at-or-above the tuned baselines", not a fixed ranking. The tree defaults pay in calibration (RF underconfident, LightGBM overconfident); Aito is best or near-best on calibration with 0 s training and a $why behind every prediction; AutoML matches Aito's calibration only after a per-target search.
• Statistics: n=200 test (95% CI ≈ ±7pp at 50%), single synthetic seed. Read the direction — which method handles the ambiguity and stays calibrated — as the durable signal; treat individual cells as point estimates.
• The baselines are given the vendor→industry join for free, so this isolates modelling + calibration + training cost, not who saw which column.
• We still report where Aito trails: RF keeps the best mean rank and top-3, v2's mean rank lags the pack, and a pre-trained tree serves a single field orders of magnitude faster than Aito's predict-time inference.

Related

• Benchmarks overview · GL coding / invoice routing · Performance