Evaluation Guide

The _evaluate endpoint measures how well an Aito query — a predict, recommend, match, similarity, estimate, or generic query — performs on your own data, without any separate testing infrastructure. It splits the data into train and test sets, hides the test rows from Aito, runs the query for each test row, and reports accuracy, ranking, and probability-quality metrics.

This guide covers how the split works, how to choose a good test set, how to read the metrics, and how to run large evaluations safely.

How evaluation works

An evaluate query has two parts:

• evaluate — the query to measure (the same body you would send to _predict, _recommend, _match, _similarity, _estimate, or _query).
• test (or testSource) — which rows to test on. Everything else becomes the training data.

For each test row, Aito simulates a database that contains only the training data, runs the query, and compares the result against the row's known answer. The metrics aggregate those per-row outcomes.

{
  "test": { "$sample": 100 },
  "evaluate": {
    "from": "products",
    "where": { "name": { "$get": "name" } },
    "predict": "category"
  },
  "select": ["accuracy", "baseAccuracy", "meanRank", "n"]
}

Choosing the test set

Evaluate runs one query per test row, so the runtime grows with the size of the test set. Pick the test rows deliberately:

• $sample — a fixed number of rows (recommended). { "$sample": 100 } evaluates 100 rows. It is deterministic by default (the same rows every run, so metrics are reproducible and cacheable); pass a seed to draw a different sample, or of to sample from a subset:

  "test": { "$sample": { "n": 100, "of": { "country": "US" }, "seed": 0 } }
  

• testSource — more control. Use where to filter or sample, limit/offset to take a window, and from to draw the test set from a separate table:

  "testSource": { "where": { "$sample": 100 } }
  "testSource": { "from": "holdout", "limit": 200 }
  

• $hash — a deterministic fraction. { "id": { "$hash": { "$mod": [10, 0] } } } selects a reproducible ~10% slice — handy for a fixed, named hold-out set.

• $index — exact rows. Pick test rows by position: { "$index": 0 }, or several under $or ({ "$or": [ { "$index": 0 }, { "$index": -1 } ] } — negative counts from the end). Useful for pinning a reproducible spot-check.

• Operator conditions. Both test/testSource wheres and the evaluate where accept comparison operators — e.g. a time cutoff { "month": { "$lte": "2024-07" } } (String columns compare lexicographically) — alongside $get bindings.

Collections (v2 tables) note: on "type": "collection" tables, evaluate supports the same selection vocabulary — field conditions, comparison operators, $index, testSource {from, where, limit}, and the sampling selectors $sample (incl. the {n, of, seed} form) / $hash / $mod. Two collection-specific details: in testSource the seeded-random pick goes as where.$sample (mutually exclusive with limit, which is a first-N cut — biased toward the oldest rows on time-ordered data), and the implicit default below caps at half the table so small collections keep a train side.

If you omit both test and testSource, evaluate defaults to { "$sample": 100 } rather than testing the entire collection, and says so in the response message. This keeps an exploratory first query fast.

Reading the metrics

Select the metrics you want with the top-level select field. The most useful ones:

Always compare accuracy against baseAccuracy: a model that beats the base rate is learning something. cases is useful for inspecting individual mistakes.

The baseline is scoped to what you evaluated

A constant condition in evaluate.where — "customer_id": "CUST-0000", or a fixed range like "month": {"$lte": 6} — defines which rows you are asking about, so the baseline is measured over those rows only. A {"$get": …} binding is per-row evidence, not a filter, and does not narrow it.

This matters most in exactly the case it is easiest to miss. Evaluating one customer inside a 128 000-row multi-tenant collection, that customer's own majority class might be 47% while the collection-wide majority is 17%. Comparing against 17% would credit the model with everything the scoping already gave you. baseSamples tells you which population the number came from, so the comparison can be checked rather than assumed.

An operator Aito cannot evaluate for scoping is reported in the response's message rather than silently ignored, so a wider-than-expected baseline always says so.

Undefined metrics

A metric that is genuinely undefined — an entropy over a baseline that ranked nothing correctly, a mean rank when no correct answer appeared at all — is returned as JSON null. It is never 0 (which would mean something quite different) and never the string "NaN".

Per-query inference config inside evaluate

The evaluate object accepts the same config as a predict — and it is applied, never silently ignored:

{
  "test": { "fold": "test" },
  "evaluate": {
    "from": "purchases",
    "where": { "supplier": { "$get": "supplier" } },
    "predict": "cost_center",
    "config": { "ai": "fast", "calibrate": true }
  }
}

• config.ai selects the inference profile for every per-row predict in the evaluation.
• config.calibrate fits a temperature on the training rows only (the fit can never leak the held-out test rows) and applies it to the evaluated predictions. When applied, the response reports the fitted tau; when it cannot be applied (e.g. too little data for the fit), the response says exactly why in message — e.g. "calibration requested but NOT applied: only 4 of the required 20 predictable sample rows". There is no silent no-op.

Time limit and partial results

Every evaluate query has a time budget, maxTime, in seconds (default 300, capped at 3600). If the budget runs out, evaluate returns the results computed so far and marks the response:

{
  "truncated": true,
  "requested": 1000,
  "message": "Evaluation hit the 300s time limit after 240 of 1000 test rows; ...",
  "n": 240,
  "accuracy": 0.71
}

Truncation is never silent — the truncated, requested, and message fields are always included when it happens, even if your select does not list them. If you hit the limit, either narrow the test set ($sample/limit), raise maxTime, or run the evaluation as a job (below).

Long evaluations: jobs and cancellation

For evaluations that take longer than a single request should, submit the query as an asynchronous job and poll for the result. Jobs can also be cancelled — which is the way to stop a long evaluation you no longer need.

Cancelling interrupts the job; the evaluation loop stops cooperatively and the job completes as cancelled. See Cancel a job in the API reference.

Worked example

Evaluate how well a chatbot predicts the response operation from a user's message, testing on a deterministic 200-row sample and inspecting the mistakes:

{
  "testSource": { "where": { "$sample": { "n": 200, "seed": 0 } } },
  "evaluate": {
    "from": "messages",
    "where": { "message": { "$get": "message" } },
    "predict": "operation"
  },
  "select": ["n", "trainSamples", "accuracy", "baseAccuracy", "meanRank", "errorCases"]
}

accuracy vs baseAccuracy tells you how much the message content helps; errorCases lists the rows Aito got wrong so you can see where it struggles.