Intent Classification for Message Routing

Voyage-3 leads on short-text classification benchmarks in 2026. BGE is the strongest open alternative. OpenAI small is cheapest and good enough if intents are well-separated.

Alternatives: BGE-large-en-v1.5, Cohere Embed v3, OpenAI text-embedding-3-small

DistilBERT fine-tuned on 20-50k labeled messages hits 93-96% macro-F1 at 30ms inference. Setfit is excellent if you only have 100-500 labels per class. Zero-shot LLMs can bootstrap before you have training data but cost 50x more per message.

Alternatives: XGBoost per-label, Setfit few-shot, Claude Haiku 4 zero-shot

Haiku 4 follows structured label definitions better than GPT-4o-mini on ambiguous messages. GPT-4o-mini is 5x cheaper and good enough if your label set is well-defined. Avoid Opus/Sonnet on the hot path - overkill for a classification task.

Alternatives: GPT-4o-mini, Gemini 2.0 Flash

OPA lets non-engineers (support ops leads) tune per-label thresholds and route rules without a deploy. Critical when routing logic changes weekly based on team capacity or new product launches.

Alternatives: Custom Python service, AWS Step Functions, Temporal

Zendesk and Salesforce dominate enterprise customer operations. Sunshine gives you custom object routing. Intercom is better for product-led companies. Do not build your own queue - you will lose 3 months to it.

Alternatives: Intercom, Front, Kustomer

Argilla for ongoing labeling by support ops. Braintrust for eval runs on each classifier change. You need weekly evals because intents drift as product features and customer base change.

Alternatives: Label-Studio, Snorkel, Scale Studio

Zero-shot LLM vs fine-tuned classifier

Zero-shot Claude Haiku 4 gets 82-88% accuracy out of the box with no training data. A fine-tuned DistilBERT on 30k labels reaches 93-96%. The fine-tuned model costs 50-100x less per inference. Start zero-shot to bootstrap labels; graduate to fine-tuned once you have enough data.

Single label vs multi-label

Single-label classifiers are simpler to train and interpret but fail on messages like 'my card was charged twice AND the app crashes on checkout' (billing + bug). Multi-label sigmoid heads handle these but require labelers to mark multiple intents consistently. Most teams should go multi-label from day one.

Per-label threshold vs global threshold

A global confidence threshold over-routes common intents and under-routes rare ones (abuse, press). Per-label thresholds tuned on ROC curves give 3-5% better routing accuracy. The tradeoff is complexity - you need to eval and tune each label separately.

New intent appears (new product launch) and classifier has no label for it

Mitigation: Monitor the proportion of messages routed to 'other/unknown'. When it rises above 3% for a week, that is a signal. Add the new label to training data, bootstrap with 200-500 labeled examples using Argilla, re-train, and deploy.

Abuse or legal-threat messages routed to regular support queue

Mitigation: Set the abuse/legal-threat threshold very low - err on false positives. A human agent quickly confirms. Add regex/keyword pre-routing for explicit threat signals ('sue', 'lawyer', 'lawsuit', slurs) as a belt-and-suspenders guardrail.

Classifier drifts as product and customer base change

Mitigation: Sample 200-500 messages per week, have support ops label them, and diff against model predictions. Re-train monthly. Show drift metrics (macro-F1 trend, per-label recall trend) on a visible dashboard.

Multi-intent messages get single-routed and the second intent gets lost

Mitigation: For multi-label predictions above threshold, fan out - create linked tickets in both queues with a shared parent. Train support agents to close linked tickets as a unit.

Non-English messages routed to teams without non-English staff

Mitigation: Route by detected language first, then by intent. If a team has no speaker of that language, translate with GPT-4o-mini or Gemini 2.0 Flash and attach the translation to the ticket. Do not drop non-English messages to the bottom of the queue.