Evaluation Guide (v2, Beta)

The _evaluate endpoint measures how well a v2 query — a predict, recommend, relate, match, estimate, or generic query — performs on your own data, with no separate testing infrastructure. It splits the data into train and test sets, hides the test rows from Aito, runs the query for each test row, and reports accuracy, ranking, and probability-quality metrics.

POST /api/v2/_evaluate dispatches by the from table's engine: collection (rep2) tables run the native v2 evaluate; classic (rep1) tables keep the v1 evaluate pipeline. This guide covers the collection behaviour.

How evaluation works

An evaluate query has two parts:

  • evaluate — the query to measure (the same body you would send to _predict, _recommend, _relate, _match, _estimate, or _query).
  • test (or testSource) — which rows to test on. Everything else becomes the training data.

For each test row, Aito evaluates against a database that contains only the training data — the test rows are hidden by a transient masked delete, so no test row can leak into its own prediction — runs the query, and compares the result against the row's known answer. The metrics aggregate those per-row outcomes.

{
  "test": { "$index": { "$mod": [5, 0] } },
  "evaluate": {
    "from": "products",
    "where": { "name": { "$get": "name" } },
    "predict": "category"
  },
  "select": ["accuracy", "baseAccuracy", "meanRank", "n"]
}

Choosing the test set

Evaluate runs one query per test row, so the runtime grows with the size of the test set. On collection tables the selectors are:

  • $index — exact rows by position. { "$index": 0 }, or several under $or ({ "$or": [ { "$index": 0 }, { "$index": 1 }, { "$index": 2 } ] }). Useful for pinning a reproducible spot-check.

  • $index + $mod — a deterministic fold (recommended). { "$index": { "$mod": [k, r] } } selects every row whose position is ≡ r (mod k) — a reproducible 1/k slice with no overlap between folds. This is the leak-free way to do k-fold cross-validation on a collection: fold r is the test set, the other k−1 folds train.

    "test": { "$index": { "$mod": [10, 0] } }

    runs a deterministic 10% hold-out; sweep r from 0..9 for full 10-fold CV.

  • testSource — a windowed or separate test set. Use where (field conditions and comparison operators) to filter, limit/offset to take a window, and from to draw the test set from a different table:

    "testSource": { "from": "holdout", "limit": 200 }
"testSource": { "where": { "month": { "$lte": "2024-07" } } }

  • Operator conditions. Both the test/testSource where and the evaluate where accept comparison operators — e.g. a time cutoff { "month": { "$lte": "2024-07" } } (String columns compare lexicographically) — alongside $get bindings.

No implicit default test set. Unlike v1 tables, a collection evaluate that omits both test and testSource is a loud 400, not a silent 100-row sample — an evaluation you didn't scope should fail rather than quietly measure something you didn't intend. The sampling selectors $sample (including the {n, of, seed} form) and $hash (with $mod) work on collections, alongside $index $mod for a deterministic fraction.

Reading the metrics

Select the metrics you want with the top-level select field. The most useful ones:

MetricMeaning
n / testSamplesNumber of test rows actually evaluated
trainSamplesAverage number of training rows per prediction
accuracyShare of test rows where the top result was correct
baseAccuracyAccuracy of always guessing the most common value — the baseline to beat
accuracyGainaccuracy - baseAccuracy
meanRankAverage position of the correct answer (0 = always top)
geomMeanPGeometric mean of the probability assigned to the correct answer — probability quality
informationGainQuality of the predicted probabilities (higher is better)
error / baseError1 - accuracy and 1 - baseAccuracy
casesPer-row detail: the test case, the top result, and whether it was correct

Always compare accuracy against baseAccuracy: a model that beats the base rate is learning something. geomMeanP is the probability-fidelity signal — with the group default (below), predictions on correlated evidence are less over-confident, which shows up here rather than in accuracy.

Inference config inside evaluate

The evaluate object accepts the same config as a predict — and it is applied, never silently ignored:

{
  "test": { "$index": { "$mod": [5, 0] } },
  "evaluate": {
    "from": "purchases",
    "where": { "supplier": { "$get": "supplier" } },
    "predict": "cost_center",
    "config": { "ai": "high", "calibrate": true }
  }
}
  • config.ai selects the inference profile for every per-row predict. v2 defaults to group (group formation), so an evaluate with no config.ai already reflects the honest-confidence behaviour of a default v2 predict — see Inference (v2) — Tuning. Set "fast" to measure plain naive Bayes, or "high" for and + group.
  • config.calibrate fits a temperature on the training rows only (the fit can never see the held-out test rows) and applies it to the evaluated predictions. When applied, the response reports the fitted tau; when it cannot be applied (e.g. too little data for the fit), the response says exactly why in message. There is no silent no-op.

Time limit and partial results

Every evaluate query has a time budget, maxTime, in seconds (default 300, capped at 3600). If the budget runs out, evaluate returns the results computed so far and marks the response:

{
  "truncated": true,
  "requested": 1000,
  "message": "Evaluation hit the 300s time limit after 240 of 1000 test rows; ...",
  "n": 240,
  "accuracy": 0.71
}

Truncation is never silent — the truncated, requested, and message fields are always included when it happens, even if your select does not list them. If you hit the limit, narrow the test set (a smaller $mod fraction or limit) or raise maxTime.

Worked example: 5-fold cross-validation

Run a deterministic 5-fold sweep and average the accuracy across folds:

{
  "test": { "$index": { "$mod": [5, 0] } },
  "evaluate": {
    "from": "messages",
    "where": { "message": { "$get": "message" } },
    "predict": "operation"
  },
  "select": ["n", "trainSamples", "accuracy", "baseAccuracy", "meanRank", "geomMeanP"]
}

Repeat with [5, 1], [5, 2], [5, 3], [5, 4] and average — every row is tested exactly once, and no test row ever trains its own prediction. accuracy vs baseAccuracy tells you how much the message content helps; geomMeanP tells you whether the confidence is well-calibrated.