Evaluation Guide (v2, Beta)

The _evaluate endpoint measures how well a v2 query — a predict, recommend, relate, match, estimate, or generic query — performs on your own data, with no separate testing infrastructure. It splits the data into train and test sets, hides the test rows from Aito, runs the query for each test row, and reports accuracy, ranking, and probability-quality metrics.

POST /api/v2/_evaluate dispatches by the from table's engine: collection (rep2) tables run the native v2 evaluate; classic (rep1) tables keep the v1 evaluate pipeline. This guide covers the collection behaviour.

How evaluation works

An evaluate query has two parts:

evaluate — the query to measure (the same body you would send to _predict, _recommend, _relate, _match, _estimate, or _query).
test (or testSource) — which rows to test on. Everything else becomes the training data.

For each test row, Aito evaluates against a database that contains only the training data — the test rows are hidden by a transient masked delete, so no test row can leak into its own prediction — runs the query, and compares the result against the row's known answer. The metrics aggregate those per-row outcomes.

{
  "test": { "$index": { "$mod": [5, 0] } },
  "evaluate": {
    "from": "products",
    "where": { "name": { "$get": "name" } },
    "predict": "category"
  },
  "select": ["accuracy", "baseAccuracy", "meanRank", "n"]
}

Choosing the test set

Evaluate runs one query per test row, so the runtime grows with the size of the test set. On collection tables the selectors are:

$index — exact rows by position. { "$index": 0 }, or several under $or ({ "$or": [ { "$index": 0 }, { "$index": 1 }, { "$index": 2 } ] }). Useful for pinning a reproducible spot-check.
$index + $mod — a deterministic fold (recommended). { "$index": { "$mod": [k, r] } } selects every row whose position is ≡ r (mod k) — a reproducible 1/k slice with no overlap between folds. This is the leak-free way to do k-fold cross-validation on a collection: fold r is the test set, the other k−1 folds train.
```
"test": { "$index": { "$mod": [10, 0] } }
```
runs a deterministic 10% hold-out; sweep r from 0..9 for full 10-fold CV.
testSource — a windowed or separate test set. Use where (field conditions and comparison operators) to filter, limit/offset to take a window, and from to draw the test set from a different table:
```
"testSource": { "from": "holdout", "limit": 200 }
"testSource": { "where": { "month": { "$lte": "2024-07" } } }
```
Operator conditions. Both the test/testSource where and the evaluate where accept comparison operators — e.g. a time cutoff { "month": { "$lte": "2024-07" } } (String columns compare lexicographically) — alongside $get bindings.

No implicit default test set. Unlike v1 tables, a collection evaluate that omits both test and testSource is a loud 400, not a silent 100-row sample — an evaluation you didn't scope should fail rather than quietly measure something you didn't intend. The sampling selectors $sample (including the {n, of, seed} form) and $hash (with $mod) work on collections, alongside $index $mod for a deterministic fraction.

Reading the metrics

Select the metrics you want with the top-level select field (omit select to get all of the scalar metrics below). They fall into three groups — did it get the answer right, were the probabilities good, and is the confidence calibrated.

Discrimination & ranking — did it pick the right answer?

Metric	Meaning
`n` / `testSamples`	Number of test rows actually evaluated
`trainSamples`	Average number of training rows per prediction
`accuracy`	Share of test rows where the top result was correct
`baseAccuracy`	Accuracy of always guessing the most common value — the baseline to beat
`accuracyGain`	`accuracy - baseAccuracy`
`meanRank`	Average position of the correct answer (1 = always top)
`mrr`	Mean reciprocal rank — `mean(1 / rank)`, the standard ranking metric (higher is better)
`coverage`	Fraction of test rows whose true answer was among the candidates at all
`error` / `baseError`	`1 - accuracy` and `1 - baseAccuracy`

Probability quality — proper scoring rules. These grade the whole probability, not just the top pick, over every test row (a truth the model rated ≈0 is penalised, not skipped):

Metric	Meaning
`logLoss`	Cross-entropy in nats — `mean(−ln p(truth))`. Lower is better. The standard "log loss."
`brierScore`	Mean squared error of the top prediction's confidence. Lower is better.
`geomMeanLikelihood`	Geometric-mean probability assigned to the truth = `exp(−logLoss)` = 1 / perplexity. In `[0, 1]`, higher is better.
`logLossSkill`	`1 − logLoss / baseLogLoss` — the fraction of the base-rate model's uncertainty removed. Higher is better (`0` = no better than guessing the base rate).

Calibration — is a stated 80% actually right 80% of the time?

Metric	Meaning
`ece`	Expected Calibration Error — the gap between confidence and accuracy, averaged over confidence bins. Lower is better (`0` = perfectly calibrated).
`reliability`	Opt-in array: one row per confidence bin (`confidence`, `accuracy`, `n`) — the reliability diagram `ece` summarises.

Always compare accuracy against baseAccuracy (a model that beats the base rate is learning something), then read logLoss/geomMeanLikelihood for probability quality and ece for calibration. With the group default (below), predictions on correlated evidence are less over-confident — that shows up as a lower ece and better logLoss, not necessarily higher accuracy.

Legacy / information-theoretic names

Aito has measured prediction quality in bits over a prior since long before the Python-tooling vocabulary settled — those names are still available as exact aliases:

Legacy (bits)	= Modern	Note
`mxe` (mean cross-entropy)	`logLoss / ln 2`	Same quantity, base-2 instead of nats
`informationGain`	`logLossSkill`, in bits	Bits saved over the base-rate model (the Kononenko–Bratko information score)
`geomMeanP`	≈ `geomMeanLikelihood`	Coverage-conditioned: averaged only over rows where the truth was a candidate, so it can slightly flatter the model — prefer `geomMeanLikelihood` / `logLoss`, which count every row

Inference config inside evaluate

The evaluate object accepts the same config as a predict — and it is applied, never silently ignored:

{
  "test": { "$index": { "$mod": [5, 0] } },
  "evaluate": {
    "from": "purchases",
    "where": { "supplier": { "$get": "supplier" } },
    "predict": "cost_center",
    "config": { "ai": "high", "calibrate": true }
  }
}

config.ai selects the inference profile for every per-row predict. v2 defaults to group (group formation), so an evaluate with no config.ai already reflects the honest-confidence behaviour of a default v2 predict — see Inference (v2) — Tuning. Set "fast" to measure plain naive Bayes, or "high" for and + group.
config.calibrate fits a temperature on the training rows only (the fit can never see the held-out test rows) and applies it to the evaluated predictions. When applied, the response reports the fitted tau; when it cannot be applied (e.g. too little data for the fit), the response says exactly why in message. There is no silent no-op.

Time limit and partial results

Every evaluate query has a time budget, maxTime, in seconds (default 300, capped at 3600). If the budget runs out, evaluate returns the results computed so far and marks the response:

{
  "truncated": true,
  "requested": 1000,
  "message": "Evaluation hit the 300s time limit after 240 of 1000 test rows; ...",
  "n": 240,
  "accuracy": 0.71
}

Truncation is never silent — the truncated, requested, and message fields are always included when it happens, even if your select does not list them. If you hit the limit, narrow the test set (a smaller $mod fraction or limit) or raise maxTime.

Worked example: 5-fold cross-validation

Run a deterministic 5-fold sweep and average the accuracy across folds:

{
  "test": { "$index": { "$mod": [5, 0] } },
  "evaluate": {
    "from": "messages",
    "where": { "message": { "$get": "message" } },
    "predict": "operation"
  },
  "select": ["n", "trainSamples", "accuracy", "baseAccuracy", "meanRank", "logLoss", "ece"]
}

Repeat with [5, 1], [5, 2], [5, 3], [5, 4] and average — every row is tested exactly once, and no test row ever trains its own prediction. accuracy vs baseAccuracy tells you how much the message content helps; logLoss grades the probabilities and ece tells you whether the confidence is calibrated.

Inference (v2) · Query Operator Reference (v2) · Improving Inference (v2)
v1 Evaluation — the classic-table pipeline, with $sample/$hash selectors and evaluate jobs.