Expense categorization (preliminary)

Task: given a purchase invoice — its vendor, tax rate, amount, line count and free-text line description — predict the expense category (GL account) it should be booked to (48 categories). This is the everyday accounts-payable automation problem: every finance team codes invoices to a chart of accounts, and most of the work is repetitive.

The realistic wrinkle is that the account is determined by what was bought, not who sent it. Big generalist vendors sell across many accounts, so a model that just memorizes "vendor → its most common category" barely clears the ~20% vendor-lookup oracle. Doing better means reading the line-item text — which identifies the invoice type, and the type determines the account — modelling the residual ambiguity, and, on top of irreducible label noise, saying how sure it is.

The data. A synthetic but realistically-structured corpus (benchmarks/gen_corpus.py, seed 41): 10,000 training invoices, a 2,000-row held-out test set, 48 categories, a ~3% base rate (most-common class), and 10% label noise (so ~90% is the accuracy ceiling and calibration is meaningful). Categories are emitted from a latent invoice-type topic model: each type sprays a bag of description keywords, so the signal accumulates across the text rather than sitting in any single feature. One corpus is consumed identically by every method — Aito reads the normalized tables and follows the vendor → industry link itself; the baselines get the same join denormalized into a flat feature table — so the comparison is fair by construction.

Methods. Aito with both engines (v1 / rep1 and v2 / rep2), a tuned scikit-learn Random Forest, LightGBM, and FLAML AutoML — all scored on the identical test set with identical metric definitions. Every method emits the same accuracy + calibration + speed schema; reproduction lives in benchmarks/. An LLM + RAG baseline (gpt-5-mini, retrieval over invoice history) is a slot that still needs re-running on this corpus — see below.

Top-1 accuracy, top-3, and rank

Vendor-lookup oracle ≈ 20% top-1; base rate ≈ 3%; 10% label noise caps accuracy near 90%. 2,000 held-out rows, which puts a 95% interval of about ±2 points on each cell. (The LLM row is a pending slot — see below.)

Read honestly — and on this hold-out, top-1 does separate the methods:

• Both Aito engines lead, and the lead is measurable. Every cell below is scored on the same rows, so the comparison is paired: what decides it is the rows where two methods disagree, not whether their intervals overlap. Against LightGBM, the strongest baseline, Aito v1 wins 185 rows that LightGBM misses and loses 110 (+3.7 pt, p < 0.0001); against Random Forest it is +6.2 pt (p < 0.0001) and against FLAML AutoML +9.7 pt (p < 0.0001). Every method lands 3–4× above the ~20% vendor oracle — they are all reading the line-item text, which is where the account signal lives — but the two that read it through learned per-feature evidence read it better.
• The two Aito engines are a tie here, and we say so. v1 and v2 differ by +0.0 pt, on 49 rows against 49: p = 1, which is a tie at this sample size. An earlier revision of this page showed v2 four points behind v1; that gap was a 200-row sample, not an engine difference, and it disappears on a hold-out ten times the size.
• Aito v1 has the best mean rank (3.3), with Aito v2 second (3.4), but RF keeps the best top-3 (90.6% vs Aito v1's 90.1%): where the ensemble is unsure it still spreads plausible mass across nearby categories.
• LightGBM stays close on top-1 but with confidence it can't back up — that is the calibration story below.

Calibration: does the confidence mean anything?

Accuracy alone hides whether a model knows when it's unsure — which is what a finance team needs to decide "auto-post" vs "route to a human". We report the expected calibration error (ECE, lower is better), the top-1 Brier score (lower is better), and the calibration gap (mean confidence − accuracy; >0 means overconfident, <0 underconfident).

This is where the methods separate:

• Aito is well-calibrated straight out of the model — no temperature scaling or post-hoc step. v2 has the smallest calibration gap of any method (gap 0.009, ECE 0.048); v1 runs mildly hot (gap 0.054, ECE 0.061) while tying v2 on top-1.
• Aito v2 also has the lowest ECE and Brier score of any method (ECE 0.048, Brier 0.141). FLAML AutoML is the best-calibrated baseline (ECE 0.081, gap 0.067) — but it only gets there after a budgeted per-target hyperparameter search (see training time); the tree defaults don't.
• Random Forest is badly underconfident (gap -0.251, the highest ECE at 0.251): its averaged-tree votes hedge, so it under-claims on the calls it gets right.
• LightGBM is overconfident (gap 0.175, ECE 0.175): its softmax outputs are sharply peaked — mean confidence 93.6% — so it asserts high confidence on predictions that are right less often. On a task with real ambiguity, that is the dangerous failure mode.

Why it matters: AP automation runs on a confidence threshold — auto-post above X, route the rest to a human. A well-calibrated model auto-posts its high-confidence slice safely; an overconfident model auto-posts its confident-but-wrong predictions straight into the ledger. Calibration, not raw top-1, is what makes the accuracy usable — and it is where Aito (with no training step) and AutoML (with a long one) pull ahead of the tree defaults.

The number that isn't accuracy: training time

Every tree/AutoML baseline needs a retraining pipeline to absorb new invoices; Aito absorbs them on insert, with a $why explanation behind every prediction. On AP coding — where a new vendor or a re-mapped account shows up weekly — "0 s and explainable" is a large part of the real total cost. Note the best-calibrated baseline, AutoML, is also the most expensive to keep current.

Predict latency

The mirror image of training time. Aito does its learning at predict time (no model to pre-fit), so a single prediction costs tens of milliseconds; the tree models, once trained, infer in well under a millisecond.

So the honest read is: for a single field, a pre-trained tree serves faster — Aito's predict-time inference is tens of ms. What Aito buys with that is no training pipeline and any field predictable on demand — which is exactly what the next number is about.

Coding a whole invoice: multi-target latency (rep1 vs rep2)

Real AP coding isn't one prediction — a clerk codes several fields per invoice: the expense category, a cost center, an approver, a handler. A tree stack needs a separately-trained model per field; Aito predicts all four from the same data with no per-target training. This is where the per-invoice latency — and the rep1 vs rep2 engine cost — actually lands. approver/handler are the interesting ones: org-scoped, high-cardinality employee links, exactly the cross-table target rep2's link priors target.

The total is the per-invoice cost of coding all four fields on one engine, with 0 s training either way. On the org-scoped employee links, rep2's link-prior machinery earns its extra latency on handler: it lifts it to 81.6% (rep2) from 68.3% (rep1), while approver stays roughly level (57.6% rep2 vs 61.0% rep1) — see the GL-coding benchmark for the rank story at depth.

How much data must I label?

The training-size sweep that used to sit here is now the section's main scaling page, because it is the one with external baselines in it:

→ How much data do I need? — top-1 by training-set size from 1 to 1 000 000 rows, this same corpus, against Random Forest, LightGBM and FLAML AutoML. At N=1 000 Aito v2 leads the field by roughly seven points with no training step; by N=100 000 LightGBM is ahead. The neighbouring question — how the engine behaves as the database grows to 10M rows — is the capacity ladder.

LLM baseline — pending on this corpus

The realistic LLM deployment for this task is retrieval-augmented: for each invoice, retrieve the most similar historical invoices (same vendor, similar text) and put them in the prompt, so gpt-5-mini copies the account from the vendor's retrieved history rather than reasoning about opaque category codes. A raw few-shot prompt (no retrieval) has nothing to reason about and scores near the floor; RAG is what makes an LLM competitive here.

A prior RAG run reached parity with the tree/Aito cluster on the previous corpus vintage, but it has not been re-run on the current line-item corpus (it needs Azure credentials and costs ~15 s per prediction), so the LLM column reads — above. The archived prior result is kept at benchmarks/results/stale-oldcorpus/llm.json; benchmarks/baselines/run_llm.py --retrieval rag regenerates it against this corpus.

How to read this

• Aito v1 and v2 tie at the top of the top-1 table (79.8% each), and both lead every baseline — LightGBM, RF and AutoML — by a paired margin that separates at this sample size (p < 0.0001 against LightGBM, the closest). The tree defaults pay in calibration (RF underconfident, LightGBM overconfident); Aito is best or near-best on calibration with 0 s training and a $why behind every prediction; AutoML matches Aito's calibration only after a per-target search.
• Statistics: n=2 000 test (95% CI ≈ ±2pp), single synthetic seed. Read the direction — which method handles the ambiguity and stays calibrated — as the durable signal; treat individual cells as point estimates.
• The baselines are given the vendor→industry join for free, so this isolates modelling + calibration + training cost, not who saw which column.
• We still report where Aito trails: RF keeps the best top-3, and a pre-trained tree serves a single field orders of magnitude faster than Aito's predict-time inference.

Related

• Benchmarks overview · GL coding / invoice routing · Speed · Footprint