bug: default domain classifier underperforms on unseen data

Summary

We benchmarked the mmBERT-32K domain classifier (llm-semantic-router/mmbert32k-intent-classifier-merged) on unseen data to understand its real-world accuracy and identify areas for improvement.

Key finding: The model scores 78.2% on MMLU (14,042 independent questions the model was never trained on). The existing evaluation on MMLU-Pro test split (~82%) is inflated because the model was trained on that same split.

Why existing evaluations showed higher numbers

The training script (ft_linear.py) trains on the MMLU-Pro test split (12,032 samples). Evaluating on that same split gives ~82% accuracy — but that's testing on training data. We validated using two independent sources:

1. MMLU (original, cais/mmlu) — 14,042 questions across 57 subjects, mapped to the same 14 categories. The model was never trained on this data.
2. Custom benchmark — 924 hand-written queries across 5 styles (academic, conversational, short, boundary, gibberish) + 70 MMLU-Pro validation samples.

Independent validation: MMLU (14,042 questions)

Domain	Accuracy	Samples
law	94.4%	1,763
history	92.5%	930
economics	91.4%	742
computer science	90.8%	412
math	89.8%	1,064
chemistry	87.1%	303
physics	85.5%	640
health	85.5%	1,417
psychology	83.6%	1,157
biology	82.8%	454
engineering	69.7%	145
other	67.6%	2,455
philosophy	53.8%	1,841
business	46.5%	719
Overall	78.2%	14,042

Note: MMLU subjects were mapped to the 14 MMLU-Pro categories (e.g., professional_law + international_law + jurisprudence → law). Unmapped subjects go to other.

Custom benchmark: accuracy by query style (994 queries)

Each query is tagged with a style to measure how the model handles different types of input:

• academic — textbook/exam-style questions (MMLU-Pro format, similar to training data)
• conversational — how real users type queries: lowercase, casual phrasing, no academic structure
• short — 1-5 word fragments or keyword-style queries (e.g., "eigenvalue", "DNA replication enzyme")
• boundary — cross-domain edge cases that could belong to multiple domains (e.g., "What is the math behind neural networks?")

Style	mmBERT-32K
academic	74.5%
short	65.8%
conversational	60.0%
boundary	58.0%

The 15-point gap between academic (74.5%) and conversational (60.0%) suggests the MMLU-Pro training data doesn't prepare the model for how real users write queries.

Key observations

1. Philosophy and business are the weakest domains on independent data (53.8% and 46.5% on MMLU). These are domains where vocabulary overlaps heavily with other categories.

2. Conversational queries score significantly lower than academic across all domains — the MMLU-Pro training data is academic exam questions but real users write casually.

3. Prompt compression compounds the accuracy gap. Per #1437, domain match after compression is 68.2%. Combined with 78.2% base accuracy, long compressed prompts may see ~53% effective domain accuracy.

4. The model was trained and evaluated on the same MMLU-Pro test split, which made accuracy appear higher (~82%) than it actually is on unseen data (78.2%).

Relationship to other work

• #1380 — Routing logic improvements (multi-category matching, real confidence scores). Complementary to model quality — both are needed.
• #1437 — Prompt compression. Domain match after compression (68.2%) compounds with base model accuracy.
• #51 — Router performance benchmark. We built a standalone benchmark tool (994 queries, ~24s on CPU) that could address this.

Benchmark tool

We built a standalone benchmark script that runs in ~24s on CPU with no LLM dependency. It supports testing any model via --model-path and filtering by query style. Will submit as a separate PR to src/training/model_eval/.

We also validated against MMLU (14K independent questions) as a cross-check. The custom benchmark and MMLU results are directionally consistent but differ in absolute numbers — the MMLU results are more trustworthy since neither side wrote that data.

H2 execution scope

Use this issue as the accuracy/calibration track for the built-in domain/category signal. It should align with #2335 and #2360 rather than only tuning one threshold in isolation.

Relevant code surfaces: src/semantic-router/pkg/classification, src/semantic-router/pkg/config/signal_config.go, src/training, and decision-level eval from #2333.

Acceptance criteria

• Unseen-domain failures are reproduced with a committed or documented eval set.
• Accuracy, false positive, false negative, and confidence calibration are reported before and after the change.
• Default domain classifier changes include threshold/model provenance and rollback notes.
• Router decisions that depend on domain signals are covered by decision-level eval or targeted tests.

Validation

make agent-report ENV=cpu CHANGED_FILES="src/semantic-router/pkg/classification src/semantic-router/pkg/config src/training website/docs"
make agent-lint CHANGED_FILES="src/semantic-router/pkg/classification src/semantic-router/pkg/config src/training website/docs"
make agent-ci-gate CHANGED_FILES="src/semantic-router/pkg/classification src/semantic-router/pkg/config src/training website/docs"