Invoice line → product matching (preliminary)

Task: an ERP receives a purchase-invoice line reading 983668 Fazer Konfektyr 1kg, 24 kpl from a supplier. The catalogue holds thousands of SKUs, a dozen of them Fazer Konfektyr variants. Which SKU is this line?

This is matching, not classification, and the difference is what the benchmark is about:

• the answer set is a live table that gains and loses rows every week, so there is no fixed label space to train on;
• every candidate has its own attributes (name, supplier, unit of measure, price) that are themselves evidence;
• the same product is billed under a different wording by every supplier, so the line text alone often cannot separate siblings.

The industry default is a lexical index (BM25 / Elasticsearch) over the catalogue, with a rules layer bolted on for the article numbers. Aito answers it as a cross-table link prediction — predict: "sku" on the invoice-line table — and that single query reads three kinds of evidence at once:

1. catalogue text — what a search index reads;
2. billing history — every wording this SKU has been invoiced under before, and by whom;
3. the line's structured fields — supplier, unit of measure, unit price, quantity — each with its own learned lift.

Accuracy against BM25

A search baseline is only honest if it reads the same evidence. BM25 is therefore run in three strengths: over the catalogue text only; over the training wordings only (each SKU as the bag of lines it was billed on); and over both, which is the fair one and the row to compare against.

The catalogue-only row is the baseline most systems ship. A lexical index over the catalogue reaches 67.9%. Aito's demo query reaches 86.1% on the same lines and the same candidates. That gap is what reading the billing history is worth.

The fair row is the honest one. Give BM25 the history too, every past wording indexed against its SKU, and it passes the demo's query at top-1: 89.9% against 86.1%.

Add $knn and Aito leads at top-1. The same query with one more where clause, "$knn": { "near": { "description": <the line> }, "k": 5, "weight": 0.5 }, lets the five invoice lines most like this one (ranked by BM25 over their own descriptions) vote for their SKU, blended into the prediction. It reaches 92.3%, which is +2.5 pt over the fair baseline on the same 2000 lines (95% interval +1.0 pt to +3.9 pt; exact McNemar p = 0.0011), and +6.3 pt over the demo's query alone.

No index is built for it. The fair BM25 row needs a document per SKU holding every wording that SKU was ever billed under, rebuilt or appended as new invoice lines arrive. The $knn clause reads the same history as rows already in the invoice_lines table: a corrected match is evidence on the next query, with nothing to reindex.

How the setting was chosen. k and weight were swept over k in {5, 10, 20, 50} and weight in {0.3, 0.5, 0.7, 1.0} on a validation split of 2,000 training lines, drawn the same way as the held-out set. The best setting was frozen and then scored once on the held-out lines. Nothing on this page was tuned on the lines it reports.

At top-5 the fair baseline is still ahead (98.5% against 96.9% with $knn, and 94.5% for the demo's query alone). If your workflow shows a person five candidates, BM25 gets the right SKU onto the shortlist slightly more often. If it assigns the first answer automatically, the $knn query is right more often.

So the claim this page supports is: with its history read as rows, Aito's first answer is the more accurate one and BM25's shortlist the more complete; BM25 needs its history-indexing layer built and maintained, and Aito's is the database.

Over the full 2000-line sweep, the basedOn setting matters too: naming the product name (what the demo now sends) reaches 86.1%, no basedOn at all 84.5%, and the supplier-only form the demo sent before 22.9 83.3%. basedOn states which of a linked row's fields a query may use as evidence (why).

Where the two disagree

Top-1 agreement between the demo's query (without $knn) and the fair BM25 baseline, over the same 2000 lines:

The two methods agree on the overwhelming majority and are close overall, but they are not interchangeable: they fail on different lines. Aito's exclusive wins are lines where the wording is the supplier's own and only the history can resolve it; BM25's are lines where the wording quotes the catalogue and popularity has pulled Aito toward a better-attested sibling. The disagreeing set is small enough here that the split between those two columns is suggestive, not decisive — the regime story is what the per-band analysis in the booktest is for.

What a win and a loss look like

Counts say how often each method wins. They do not say which half of your workload lands where — so here are the actual lines, taken from the measured set in order rather than chosen, with each SKU's training billing count beside it. Warmth is the variable the whole comparison turns on, so it is printed rather than described.

Both right — the bulk of the work. The supplier quotes enough of the catalogue name that the text settles it. Neither method is doing anything clever, and on a workload made only of these lines a search index is the right tool.

Aito right, BM25 wrong — what the history buys. The wording names a family and the catalogue holds several members; the text cannot choose between them. What chooses is who is billing, at what unit and price — fields a lexical index has no way to use, because they are not text to match against. Look at what BM25 answers instead: never something unrelated, always a near-identical variant — a different Mk, a different pack size, the same product from a different warehouse. The text genuinely does not separate them, and the fair baseline has nothing left to break the tie with.

Aito wrong, BM25 right — where we lose. Two different failures sit in this cell, and they need different fixes. Most are warm-candidate domination: the text names a thinly-billed SKU and Aito answers with a far more familiar sibling, confidently. The rest are attribute confusions between two comparably-billed variants, where a colour or a pack size got dropped — the opposite of a popularity problem.

Neither — the task's own floor. Truncated and abbreviated wordings over a catalogue of near-identical variants. Both methods often land on the same wrong SKU, which is the tell: the line does not contain what would separate them. Lines like these are why the workflow needs a human reviewer and why $why earns its latency — no ranking change will fix a line that does not say which warehouse it means.

Where Aito is still weak

Two things on this corpus, stated plainly.

Warm-candidate domination. There are 222 lines that the catalogue-only index — a model with no history at all — gets right and Aito gets wrong. In most of them the text names a thinly-billed SKU and Aito answers with a sibling it has seen ten to twenty times as often (T-shirt S Navy Pro, billed 5 times, loses to T-shirt S Navy, billed 86 — one row of the worked-example tables the same booktest emits, which list every line in this cell with both billing counts). It is a real defect and the one we are working on: a correct SKU seen five times should not lose to a sibling seen five hundred when the text names the first one. The remainder of that cell is a different problem — confusions between two comparably-billed variants, where a colour or a pack size is what got dropped, and occasionally a pair whose catalogue names are identical apart from a warehouse the invoice line never mentions.

No history, no advantage. Every gain on this page comes from the billing history. On public entity-matching corpora where the gold pairs have no transaction history at all — two product catalogues to be reconciled, nothing ever bought — Aito has no history to read and measures well behind BM25. Those runs are not in this page's metrics pipeline and no figures are quoted for them, but the direction is not in doubt and the reason is structural, not a tuning gap. If your matching problem is a one-off reconciliation of two static lists, use a search engine. If it is a recurring flow where the same things get matched over and over and people correct the results, that is the case this page measures.

Latency

The demo's exact query — five where conditions, $why on, limit 5 — against the full catalogue:

There are three regimes here and only one of them is the number to plan with. The run is one open database with one view shared by every query, over the same 2000 distinct lines as the accuracy tables — so it warms as it goes, and the table says so instead of averaging it away:

• The first request pays 5448 ms while the caches fill. That is what you will see if you open a fresh database and send one query — more than an order of magnitude above the steady state, so a deployment that matters should warm up rather than let a user meet it.
• Warming takes longer than one request. The first 25 average 448 ms, against 266 ms for the last 25.
• 266 ms is the serving figure — the regime a live deployment runs in, and the one to size capacity against. The mean over all 2000 is higher (314 ms) precisely because it still carries the ramp; it is reported here rather than quietly dropped.

These are single-node, single-threaded figures with $why on — asking for the explanation is part of the cost.

For implementers

If you are building this on Aito, the shape below is the one that produced every number above.

Schema. The catalogue is one collection; the transaction collection links to it. That link is what makes this a prediction rather than a search.

PUT /api/v2/schema/products
{
  "type": "collection",
  "columns": {
    "sku":             { "type": "String" },
    "name":            { "type": "Text", "analyzer": "english" },
    "supplier":        { "type": "String" },
    "category":        { "type": "String" },
    "unit_of_measure": { "type": "String" },
    "unit_price":      { "type": "Decimal" }
  }
}

PUT /api/v2/schema/invoice_lines
{
  "type": "collection",
  "columns": {
    "description":      { "type": "Text", "analyzer": "english" },
    "billing_supplier": { "type": "String" },
    "unit_of_measure":  { "type": "String" },
    "unit_price_eur":   { "type": "Decimal" },
    "quantity":         { "type": "Int" },
    "sku":              { "type": "String", "link": "products.sku" }
  }
}

The query. Put the whole incoming line in where — the free text and the structured fields — and predict the link column.

POST /api/v2/_predict
{
  "from": "invoice_lines",
  "where": {
    "description": "983668 Fazer Konfektyr 1kg, 24 kpl",
    "billing_supplier": "Baltic Trade House",
    "unit_of_measure": "kg",
    "unit_price_eur": 12.40,
    "quantity": 24
  },
  "predict": "sku",
  "config": { "ai": "and" },
  "select": ["$p", "$value", "$why"],
  "limit": 5
}

Four things that matter, each measured:

• Feed every field, not just the text. The structured fields carry real lift — supplier especially, because who bills you narrows the catalogue before a single token is read.
• config.ai: "and" treats co-occurring tokens as joint evidence rather than independent votes. It is what makes a multi-token wording behave like a phrase.
• Add $knn over the line's own text. "$knn": { "near": { "description": <the line> }, "k": 5, "weight": 0.5 } in the where lets the most similar past invoice lines vote, and it is the largest single gain on this page (see accuracy). It reads the billing history as rows; there is no index to build.
• Name the product name as evidence. basedOn: ["name"], what the demo sends, measures best of the three settings in the table below. basedOn states which of the linked product's fields a query may use, so a setting that leaves out the name scores the line only against the billing history.
• Write the corrections back. Every accepted match becomes a training row the next query reads, with no reindex and no fit step. That is the loop the whole advantage rests on — a deployment that never writes back its corrections is the catalogue-only row, not the Aito row.

Measured over the full 2000-line hold-out, each with the demo's vendor clauses and without $knn:

Warm the database before you serve from it. The first request pays 5448 ms and the ramp runs past the first two dozen — see latency and the warming strategies guide. This is a deployment detail, not a tuning one, and it is the most common reason a first trial feels slower than this page.

Ask for $why. Each hit carries the factors that produced its probability — which tokens fired, which structured field lifted it, the base rate it started from. It costs latency (the numbers above include it) and it is what makes a human reviewer's correction an informed one rather than a coin flip.

Read honestly

• The comparison that flatters us is the wrong one. Against a catalogue-only search index the margin is large; against BM25 given the same history it is small. The second is the real comparison, and the second is what the page leads with.
• The advantage is history, not ranking. Take the history away and the advantage goes with it — see above. This benchmark measures a recurring flow with feedback, which is the case Aito is for; it does not claim anything about static list reconciliation.
• Warm candidates still beat thin correct ones on most of the 222 lines a history-free index gets right and we do not. Open defect, not a tuning knob.
• The corpus is generated, one seed. It comes from the aito-erp-demo generator, which models real supplier behaviour — each supplier writes the same product its own way, some quote the catalogue, some don't — but it is not a customer's data, and a generator cannot surprise you the way real wording does. n = 2000 for the head-to-head (the query-setting table runs the full 2000-line hold-out). At that size a point or two is inside the noise; read the direction and the size of the catalogue-only gap, not the last digit.
• The numbers are produced, not typed. The booktest behind them refuses to publish a short run, or a run on a loaded machine, so a truncated diagnostic pass cannot quietly overwrite what this page shows.

Related

• Benchmarks overview · GL coding / invoice routing · Query, predict & write speed · Inference (v2) · Improving inference