Invoice line → product matching (Beta)

Task: an ERP receives a purchase-invoice line reading 983668 Fazer Konfektyr 1kg, 24 kpl from a supplier. The catalogue holds thousands of SKUs, a dozen of them Fazer Konfektyr variants. Which SKU is this line?

This is matching, not classification, and the difference is what the benchmark is about:

• the answer set is a live table that gains and loses rows every week, so there is no fixed label space to train on;
• every candidate has its own attributes (name, supplier, unit of measure, price) that are themselves evidence;
• the same product is billed under a different wording by every supplier, so the line text alone often cannot separate siblings.

The industry default is a lexical index (BM25 / Elasticsearch) over the catalogue, with a rules layer bolted on for the article numbers. Aito answers it as a cross-table link prediction — predict: "sku" on the invoice-line table — and that single query reads three kinds of evidence at once:

1. catalogue text — what a search index reads;
2. billing history — every wording this SKU has been invoiced under before, and by whom;
3. the line's structured fields — supplier, unit of measure, unit price, quantity — each with its own learned lift.

Accuracy against BM25

A search baseline is only honest if it reads the same evidence. BM25 is therefore run in three strengths: over the catalogue text only; over the training wordings only (each SKU as the bag of lines it was billed on); and over both, which is the fair one and the row to compare against.

The catalogue-only row is the headline. A lexical index over the catalogue — the thing most matching systems actually ship — reaches 64.4%. Aito reaches 90.4% on the same lines and the same candidates. The gap is what reading the billing history is worth, and it is the largest effect on this page.

The fair row is the honest one. Give BM25 the history too — index every past wording against its SKU — and it closes most of that gap, to 89.2%. Aito is still ahead at top-1, but by around a point: read that as "at least parity with a well-built lexical baseline", not as a rout. At top-5 the fair baseline is ahead (98.8% vs 98.0%) — Aito converts more of its shortlist into a correct first answer, BM25 gets the right SKU into the shortlist slightly more often. If your workflow shows a human five candidates, that difference matters more than the top-1 row.

So the claim this page supports is not "Aito is a better ranker than BM25". It is: BM25 needs the history-indexing layer built, maintained and rebuilt; Aito's is the database.

Where the two disagree

Top-1 agreement against the fair BM25 baseline, over the same 250 lines:

The two methods agree on the overwhelming majority and are close overall, but they are not interchangeable: they fail on different lines. Aito's exclusive wins are lines where the wording is the supplier's own and only the history can resolve it; BM25's are lines where the wording quotes the catalogue and popularity has pulled Aito toward a better-attested sibling. The disagreeing set is small enough here that the split between those two columns is suggestive, not decisive — the regime story is what the per-band analysis in the booktest is for.

What a win and a loss look like

Counts say how often each method wins. They do not say which half of your workload lands where — so here are the actual lines, taken from the measured set in order rather than chosen, with each SKU's training billing count beside it. Warmth is the variable the whole comparison turns on, so it is printed rather than described.

Both right — the bulk of the work. The supplier quotes enough of the catalogue name that the text settles it. Neither method is doing anything clever, and on a workload made only of these lines a search index is the right tool.

Aito right, BM25 wrong — what the history buys. The wording names a family and the catalogue holds several members; the text cannot choose between them. What chooses is who is billing, at what unit and price — fields a lexical index has no way to use, because they are not text to match against. Look at what BM25 answers instead: never something unrelated, always a near-identical variant — a different Mk, a different pack size, the same product from a different warehouse. The text genuinely does not separate them, and the fair baseline has nothing left to break the tie with.

Aito wrong, BM25 right — where we lose. Two different failures sit in this cell, and they need different fixes. Most are warm-candidate domination: the text names a thinly-billed SKU and Aito answers with a far more familiar sibling, confidently. The rest are attribute confusions between two comparably-billed variants, where a colour or a pack size got dropped — the opposite of a popularity problem.

Neither — the task's own floor. Truncated and abbreviated wordings over a catalogue of near-identical variants. Both methods often land on the same wrong SKU, which is the tell: the line does not contain what would separate them. Lines like these are why the workflow needs a human reviewer and why $why earns its latency — no ranking change will fix a line that does not say which warehouse it means.

Where Aito is still weak

Two things on this corpus, stated plainly.

Warm-candidate domination. There are 18 lines that the catalogue-only index — a model with no history at all — gets right and Aito gets wrong. In most of them the text names a thinly-billed SKU and Aito answers with a sibling it has seen ten to twenty times as often (T-shirt S Navy Pro, billed 5 times, loses to T-shirt S Navy, billed 86 — one row of the worked-example tables the same booktest emits, which list every line in this cell with both billing counts). It is a real defect and the one we are working on: a correct SKU seen five times should not lose to a sibling seen five hundred when the text names the first one. The remainder of that cell is a different problem — confusions between two comparably-billed variants, where a colour or a pack size is what got dropped, and occasionally a pair whose catalogue names are identical apart from a warehouse the invoice line never mentions.

No history, no advantage. Every gain on this page comes from the billing history. On public entity-matching corpora where the gold pairs have no transaction history at all — two product catalogues to be reconciled, nothing ever bought — Aito has no history to read and measures well behind BM25. Those runs are not in this page's metrics pipeline and no figures are quoted for them, but the direction is not in doubt and the reason is structural, not a tuning gap. If your matching problem is a one-off reconciliation of two static lists, use a search engine. If it is a recurring flow where the same things get matched over and over and people correct the results, that is the case this page measures.

Latency

The demo's exact query — five where conditions, $why on, limit 5 — against the full catalogue:

There are three regimes here and only one of them is the number to plan with. The run is one open database with one view shared by every query, over the same 250 distinct lines as the accuracy tables — so it warms as it goes, and the table says so instead of averaging it away:

• The first request pays 3380 ms while the caches fill. That is what you will see if you open a fresh database and send one query — more than an order of magnitude above the steady state, so a deployment that matters should warm up rather than let a user meet it.
• Warming takes longer than one request. The first 25 average 344 ms, against 199 ms for the last 25.
• 199 ms is the serving figure — the regime a live deployment runs in, and the one to size capacity against. The mean over all 250 is higher (215 ms) precisely because it still carries the ramp; it is reported here rather than quietly dropped.

These are single-node, single-threaded figures with $why on — asking for the explanation is part of the cost.

For implementers

If you are building this on Aito, the shape below is the one that produced every number above.

Schema. The catalogue is one collection; the transaction collection links to it. That link is what makes this a prediction rather than a search.

PUT /api/v2/schema/products
{
  "type": "collection",
  "columns": {
    "sku":             { "type": "String" },
    "name":            { "type": "Text", "analyzer": "english" },
    "supplier":        { "type": "String" },
    "category":        { "type": "String" },
    "unit_of_measure": { "type": "String" },
    "unit_price":      { "type": "Decimal" }
  }
}

PUT /api/v2/schema/invoice_lines
{
  "type": "collection",
  "columns": {
    "description":      { "type": "Text", "analyzer": "english" },
    "billing_supplier": { "type": "String" },
    "unit_of_measure":  { "type": "String" },
    "unit_price_eur":   { "type": "Decimal" },
    "quantity":         { "type": "Int" },
    "sku":              { "type": "String", "link": "products.sku" }
  }
}

The query. Put the whole incoming line in where — the free text and the structured fields — and predict the link column.

POST /api/v2/_predict
{
  "from": "invoice_lines",
  "where": {
    "description": "983668 Fazer Konfektyr 1kg, 24 kpl",
    "billing_supplier": "Baltic Trade House",
    "unit_of_measure": "kg",
    "unit_price_eur": 12.40,
    "quantity": 24
  },
  "predict": "sku",
  "config": { "ai": "and" },
  "select": ["$p", "$value", "$why"],
  "limit": 5
}

Four things that matter, each measured:

• Feed every field, not just the text. The structured fields carry real lift — supplier especially, because who bills you narrows the catalogue before a single token is read.
• config.ai: "and" treats co-occurring tokens as joint evidence rather than independent votes. It is what makes a multi-token wording behave like a phrase.
• Do not add the catalogue name as candidate evidence. The obvious tuning move is basedOn: ["name"], so the product's own name scores against the line. On this corpus it costs about ten points of top-1 (table below). With no basedOn, every linked sub-feature is already used; naming one narrows the evidence rather than adding to it.
• Write the corrections back. Every accepted match becomes a training row the next query reads, with no reindex and no fit step. That is the loop the whole advantage rests on — a deployment that never writes back its corrections is the catalogue-only row, not the Aito row.

Measured over the full 2000-line hold-out:

Warm the database before you serve from it. The first request pays 3380 ms and the ramp runs past the first two dozen — see latency and the warming strategies guide. This is a deployment detail, not a tuning one, and it is the most common reason a first trial feels slower than this page.

Ask for $why. Each hit carries the factors that produced its probability — which tokens fired, which structured field lifted it, the base rate it started from. It costs latency (the numbers above include it) and it is what makes a human reviewer's correction an informed one rather than a coin flip.

Read honestly

• The comparison that flatters us is the wrong one. Against a catalogue-only search index the margin is large; against BM25 given the same history it is small. The second is the real comparison, and the second is what the page leads with.
• The advantage is history, not ranking. Take the history away and the advantage goes with it — see above. This benchmark measures a recurring flow with feedback, which is the case Aito is for; it does not claim anything about static list reconciliation.
• Warm candidates still beat thin correct ones on most of the 18 lines a history-free index gets right and we do not. Open defect, not a tuning knob.
• The corpus is generated, one seed. It comes from the aito-erp-demo generator, which models real supplier behaviour — each supplier writes the same product its own way, some quote the catalogue, some don't — but it is not a customer's data, and a generator cannot surprise you the way real wording does. n = 250 for the head-to-head (the query-setting table runs the full 2000-line hold-out). At that size a point or two is inside the noise; read the direction and the size of the catalogue-only gap, not the last digit.
• The numbers are produced, not typed. The booktest behind them refuses to publish a short run, or a run on a loaded machine, so a truncated diagnostic pass cannot quietly overwrite what this page shows.

Related

• Benchmarks overview · GL coding / invoice routing · Query, predict & write speed · Inference (v2) · Improving inference