Evaluation Guide

The _evaluate endpoint measures how well an Aito query — a predict, recommend, match, similarity, estimate, or generic query — performs on your own data, without any separate testing infrastructure. It splits the data into train and test sets, hides the test rows from Aito, runs the query for each test row, and reports accuracy, ranking, and probability-quality metrics.

This guide covers how the split works, how to choose a good test set, how to read the metrics, and how to run large evaluations safely.

How evaluation works

An evaluate query has two parts:

evaluate — the query to measure (the same body you would send to _predict, _recommend, _match, _similarity, _estimate, or _query).
test (or testSource) — which rows to test on. Everything else becomes the training data.

For each test row, Aito simulates a database that contains only the training data, runs the query, and compares the result against the row's known answer. The metrics aggregate those per-row outcomes.

{
  "test": { "$sample": 100 },
  "evaluate": {
    "from": "products",
    "where": { "name": { "$get": "name" } },
    "predict": "category"
  },
  "select": ["accuracy", "baseAccuracy", "meanRank", "n"]
}

Choosing the test set

Evaluate runs one query per test row, so the runtime grows with the size of the test set. Pick the test rows deliberately:

$sample — a fixed number of rows (recommended). { "$sample": 100 } evaluates 100 rows. It is deterministic by default (the same rows every run, so metrics are reproducible and cacheable); pass a seed to draw a different sample, or of to sample from a subset:
```
"test": { "$sample": { "n": 100, "of": { "country": "US" }, "seed": 0 } }
```
testSource — more control. Use where to filter or sample, limit/offset to take a window, and from to draw the test set from a separate table:
```
"testSource": { "where": { "$sample": 100 } }
"testSource": { "from": "holdout", "limit": 200 }
```
$hash — a deterministic fraction. { "id": { "$hash": { "$mod": [10, 0] } } } selects a reproducible ~10% slice — handy for a fixed, named hold-out set.
$index — exact rows. Pick test rows by position: { "$index": 0 }, or several under $or ({ "$or": [ { "$index": 0 }, { "$index": -1 } ] } — negative counts from the end). Useful for pinning a reproducible spot-check.
Operator conditions. Both test/testSource wheres and the evaluate where accept comparison operators — e.g. a time cutoff { "month": { "$lte": "2024-07" } } (String columns compare lexicographically) — alongside $get bindings.

Collections (v2 tables) note: on "type": "collection" tables, evaluate supports the same selection vocabulary — field conditions, comparison operators, $index, testSource {from, where, limit}, and the sampling selectors $sample (incl. the {n, of, seed} form) / $hash / $mod. Two collection-specific details: in testSource the seeded-random pick goes as where.$sample (mutually exclusive with limit, which is a first-N cut — biased toward the oldest rows on time-ordered data), and the implicit default below caps at half the table so small collections keep a train side.

If you omit both test and testSource, evaluate defaults to { "$sample": 100 } rather than testing the entire collection, and says so in the response message. This keeps an exploratory first query fast.

Reading the metrics

Select the metrics you want with the top-level select field. The most useful ones:

Metric	Meaning
`n` / `testSamples`	Number of test rows actually evaluated
`trainSamples`	Average number of training rows per prediction
`accuracy`	Share of test rows where the top result was correct
`baseAccuracy`	Accuracy of always guessing the most common value — the baseline to beat
`accuracyGain`	`accuracy - baseAccuracy`
`meanRank`	Average position of the correct answer (0 = always top)
`informationGain`	Quality of the predicted probabilities (higher is better)
`error` / `baseError`	`1 - accuracy` and `1 - baseAccuracy`
`rmse`, `mae`, `r2`	Regression metrics, for `estimate` evaluations
`cases`	Per-row detail: the test case, the top result, and whether it was correct

Always compare accuracy against baseAccuracy: a model that beats the base rate is learning something. cases is useful for inspecting individual mistakes.

Per-query inference config inside evaluate

The evaluate object accepts the same config as a predict — and it is applied, never silently ignored:

{
  "test": { "fold": "test" },
  "evaluate": {
    "from": "purchases",
    "where": { "supplier": { "$get": "supplier" } },
    "predict": "cost_center",
    "config": { "ai": "fast", "calibrate": true }
  }
}

config.ai selects the inference profile for every per-row predict in the evaluation.
config.calibrate fits a temperature on the training rows only (the fit can never leak the held-out test rows) and applies it to the evaluated predictions. When applied, the response reports the fitted tau; when it cannot be applied (e.g. too little data for the fit), the response says exactly why in message — e.g. "calibration requested but NOT applied: only 4 of the required 20 predictable sample rows". There is no silent no-op.

Time limit and partial results

Every evaluate query has a time budget, maxTime, in seconds (default 300, capped at 3600). If the budget runs out, evaluate returns the results computed so far and marks the response:

{
  "truncated": true,
  "requested": 1000,
  "message": "Evaluation hit the 300s time limit after 240 of 1000 test rows; ...",
  "n": 240,
  "accuracy": 0.71
}

Truncation is never silent — the truncated, requested, and message fields are always included when it happens, even if your select does not list them. If you hit the limit, either narrow the test set ($sample/limit), raise maxTime, or run the evaluation as a job (below).

Long evaluations: jobs and cancellation

For evaluations that take longer than a single request should, submit the query as an asynchronous job and poll for the result. Jobs can also be cancelled — which is the way to stop a long evaluation you no longer need.

Step	Request
Start the evaluation	`POST /api/v1/jobs/_evaluate` with the evaluate body
Check status	`GET /api/v1/jobs/{id}`
Fetch the result	`GET /api/v1/jobs/{id}/result`
Cancel a running job	`DELETE /api/v1/jobs/{id}`

Cancelling interrupts the job; the evaluation loop stops cooperatively and the job completes as cancelled. See Cancel a job in the API reference.

Worked example

Evaluate how well a chatbot predicts the response operation from a user's message, testing on a deterministic 200-row sample and inspecting the mistakes:

{
  "testSource": { "where": { "$sample": { "n": 200, "seed": 0 } } },
  "evaluate": {
    "from": "messages",
    "where": { "message": { "$get": "message" } },
    "predict": "operation"
  },
  "select": ["n", "trainSamples", "accuracy", "baseAccuracy", "meanRank", "errorCases"]
}

accuracy vs baseAccuracy tells you how much the message content helps; errorCases lists the rows Aito got wrong so you can see where it struggles.