Smart search & recommendations (Beta)

Task: a shopper types a query and is shown a list of products. Rank that list so the product they went on to buy comes first. Then do the same with no query at all: the For You shelf, ranked from who the shopper is.

Aito answers both with one recommend query over the shop's own impression log: rank the products shown, by the probability they get bought, given the query words, the shopper's profile and the products' attributes. There is no model to train and no search index to sync.

The data

No public dataset carries everything a smart-search ranker learns from: real query wordings, who searched, and what they bought. Amazon ESCI has queries but no shoppers, MIND has shoppers but no queries, and Diginetica's tokens are anonymized. So this benchmark uses a generated musical-instrument shop whose answer key is known.

A purchase comes from a hidden model with three parts:

• intent: what the shopper is in the market for;
• user fit: skill level, budget, favourite brand and instrument;
• season: what sells when.

The query is a lossy wording of the intent. Some shoppers type the catalogue's own words ("electric guitar"), some a concept ("beginner guitar"), some a brand, some slang the catalogue never uses ("axe"), some run the words together, and a few make typos.

Because the generator records every candidate's true score, each result is read against oracles computed on the same searches. An oracle ranks by the true score, so it is the most a ranker could reach from that information. The total oracle is the ceiling for the task.

Scoring uses 300 held-out searches, each with the 40 products actually shown, and nDCG@10 with the bought product as the one relevant item.

Result

How to read it.

• Plain text search is a real baseline, and Aito's own ranks like BM25. $similarity produces the same ranking as the reference BM25 on 280 of the 300 searches.
• The strongest BM25 row is the fair one. It indexes, for each product, every past query it was bought under: the same history Aito learns from, handed to a lexical index. A learned ranker only earns its keep where it beats that row.
• basedOn is what makes the difference. It is recommend's way of scoring products through their attributes (category, subcategory, brand, price tier, skill level). Without it, a query word only says something about the exact products it was bought with. With it, the word's evidence reaches every product in the same subcategory, including ones no shopper has searched that way yet.
• The hybrids are tuned on a separate set of searches, never on the scored ones.

Is the difference real?

Every ranker is scored on the same searches, so each comparison is a per-search difference with a 95% bootstrap interval. An interval that contains zero means these searches do not separate the two rankers.

So recommend with basedOn beats both the fair BM25 index and the older hybrid. Adding the shopper profile, or blending BM25 back in, does not measurably change it on these searches.

By how the shopper phrased it

• Where text search wins: brand queries. The brand name is in the product text, so an exact token match finds the brand directly.
• Where learned evidence wins: every wording the catalogue does not contain. Slang and run-together words share no token with the product text, so a lexical index over it cannot rank them. recommend learns which products those words led to, and basedOn carries that to the whole subcategory.
• The run-together and typo rows have few searches each. Read them as direction, not digits.

For You: no query at all

Against popularity, the profile + basedOn shelf gains +0.143 nDCG@10 (95% interval +0.106 to +0.180). It can sit above the user-fit oracle without leaking the answer. That oracle scores user fit alone, while a shopper's profile also hints at intent: people mostly shop for their own instrument.

The query

The smart-search ranking is one request against the impression log. goal is the outcome to rank for, where is what is known about this search, and basedOn names the product attributes the evidence should flow through.

{
  "from": "impressions",
  "where": {
    "search.query": "axe for a beginner",
    "search.user_skill": "beginner",
    "search.user_budget": "budget"
  },
  "recommend": "product",
  "goal": { "purchase": true },
  "basedOn": ["category", "subcategory", "brand", "tier", "skill_level"],
  "select": ["id", "name", "$p", "$why"],
  "limit": 10
}

For the For You shelf, drop search.query and keep the profile. $why breaks each $p down into the evidence behind it: which query word, which profile value, and through which attribute.

Honest limits

• A generated corpus, one seed. It is designed so neither text nor profile is enough on its own, and its properties are checked by corpus diagnostics. It is still a model of a shop, not a shop. Real query logs are messier, and the relative standing of text search can differ.
• Profile evidence needs basedOn. Without it, adding the shopper profile to a query lowers the ranking: 0.413 against the query's 0.452. Per-product evidence for each profile value is thin, and it overlaps what the query already says. Give recommend the product attributes when you combine the two.
• No latency here. This suite ran in-process on a shared machine; see Query, predict & write speed for measured latency.
• Ties break by product id. The generator lists the bought product early in the shown list, so any ranker that falls back to display order is inflated; every ranker here, the baselines included, breaks ties by id.

Related

• Benchmarks overview · Full-text search (v2) · Query reference · Inference (v2)