Intent classification — Banking77 (Beta)

Task: classify a banking customer-service query into one of 77 fine-grained intents (e.g. card_arrival, pending_top_up, wrong_amount_of_cash_received). Banking77 is a standard, public NLP benchmark — 13,083 queries, 10,003 train / 3,080 test — and it's genuinely hard: 77 classes, many of them near-synonyms.

Aito predicts the intent from the query text alone, as a predict over the category field, run by the Banking77Evaluation booktest (which auto-downloads the dataset on first run).

Result

Test setAito top-1 accuracy
Full test (3,080 queries)78.4%
1,000-query subset74.9%
200-query subset80.0%

~78.4% on 77-way intent classification, with zero training — new labelled examples are usable the moment they're inserted, and every prediction comes with a $why explanation.

In context

This is not a win against purpose-built NLP models — and that's the honest picture. Fine-tuned dual sentence-encoders reach the low-90s on Banking77 in the literature that introduced it (ConveRT ≈ 93.0%, USE ≈ 92.8%, BERT ≈ 87.2%; Casanueva et al., 2020). Aito trails those.

What Aito trades for the gap is the entire training-and-serving apparatus behind them:

  • No fine-tuning, no GPU. The transformer numbers come from pre-trained sentence encoders fine-tuned on the task; Aito's come from predict over the raw text with no model to train or host.
  • Incremental. A new intent or a batch of freshly-labelled tickets is live on insert — no retraining cycle.
  • Explainable. $why shows which tokens drove the predicted intent.

So the read is: for a support-triage system where you can stand up a fine-tuned transformer and keep it retrained, those models win on raw accuracy. Where you want intent routing on your live data with no ML pipeline, ~79% zero-training on a 77-way problem is a strong, immediately-available baseline — and it improves as you label more, with no retraining step.

Generated. Aito's figures are interpolated at docs-build time from the Banking77Evaluation booktest snapshots, so they update on re-baseline and can't silently drift. The transformer figures are cited from the published Banking77 paper (Casanueva et al., 2020). A rep2 (v2 engine) run and a direct FLAML AutoML comparison on the same split are follow-ups.