research: define next-generation router base model and model family

Summary

Research the next-generation router base model strategy and define a new vLLM Semantic Router model family around it. The target family should cover rerankers, embeddings, and classifiers while improving toward SOTA quality under router-specific production constraints: high concurrency, long context, low latency, predictable memory use, and deployability across CPU, NVIDIA, and AMD paths.

Parent roadmap: #2287 Related signal/model roadmap: #2335, #2360 Related issues: #1485, #2247, #2250, #2251, #2252, #2301, #2347, #2372, #2379, #1510, #2394 Current base model reference: https://huggingface.co/llm-semantic-router/mmbert-32k-yarn Current classifier collection to cover/replace: https://huggingface.co/collections/llm-semantic-router/mom-multilingual-class

Motivation

The current model family is built around mmBERT/mmBERT-32K assets. Repo defaults bind mmBERT-derived models into embedding, prompt-guard, domain, PII, fact-check, and feedback modules, and docs present them as the current MoM model family.

The current mom-multilingual-class collection covers long-context multilingual classifier assets for domain, jailbreak, PII, factual, feedback, and modality-style routing tasks. H2 needs a cleaner next-generation answer: a base-model strategy plus a coherent published model family, not a one-off model replacement.

The current Hugging Face card for llm-semantic-router/mmbert-32k-yarn describes a 32K-context ModernBERT-derived model and also notes important production constraints, including long-range retrieval degradation beyond shorter distances and high memory requirements for full 32K contexts. That makes this a research, benchmarking, and migration track rather than a blind default-model swap.

Affected code and surfaces

• src/semantic-router/pkg/config/canonical_defaults.go
• src/semantic-router/pkg/config/registry*.go
• src/semantic-router/pkg/classification/**
• src/semantic-router/pkg/modelruntime/**
• src/semantic-router/pkg/modelinventory/**
• src/semantic-router/pkg/apiserver/route_model_info*.go
• src/semantic-router/pkg/vectorstore/**
• src/semantic-router/pkg/cache/**
• candle-binding/**, onnx-binding/**, nlp-binding/**, ml-binding/**
• src/training/**
• config/**, deploy/**, e2e/**, website/docs/**

Scope

• Inventory every current mmBERT/mmBERT-32K dependency across defaults, configs, model registry metadata, docs, E2E fixtures, ONNX/Candle/native bindings, runtime APIs, and Hugging Face collections.
• Establish baseline metrics for the current base model and derived signal models:
  • domain/category routing;
  • PII detection;
  • jailbreak/prompt-injection detection;
  • fact-check / grounding signals;
  • feedback detection;
  • modality/image-routing signals where applicable;
  • embedding, semantic-cache, retrieval, tool-similarity, and rerank workloads.
• Define the target next-generation model family:
  • Reranker family: cross-encoder or equivalent rerankers for two-stage retrieval/routing, KB/tool/model selection, and optional signal extraction.
  • Embedding family: long-context, low-latency embedding models for semantic cache, retrieval, tools similarity, model/profile matching, and Matryoshka or early-exit variants when useful.
  • Classifier family: multilingual/long-context classifiers that cover and can replace the current mom-multilingual-class collection, including domain/intent, jailbreak, PII, fact-check, feedback, modality, and future registered signal classifiers.
• Define a candidate model matrix for next-generation router base models, including long-context encoders, multilingual/code-aware encoders, compact latency-first encoders, and distillation/Matryoshka variants.
• Benchmark candidates on router-specific metrics:
  • signal accuracy, calibration, OOD behavior, and multilingual robustness;
  • long-context quality at realistic prompt lengths and hard long-range cases;
  • rerank quality and ranking stability;
  • p50/p95/p99 latency, throughput, batch behavior, and queue impact;
  • memory footprint under high concurrency and multiple router-model serving;
  • CPU, CUDA, ROCm, ONNX Runtime, Candle, and native binding compatibility;
  • artifact size, startup time, warmup behavior, and model-cache behavior.
• Define training/fine-tuning/distillation/quantization/export recipes for top candidates.
• Define model-family publication requirements:
  • naming/versioning conventions for reranker, embedding, and classifier artifacts;
  • model cards, dataset provenance, eval reports, and license compatibility;
  • Hugging Face collection layout and migration notes from current collections;
  • catalog entries and capability metadata;
  • threshold/calibration migration;
  • rollback plan;
  • compatibility matrix;
  • shadow-mode or opt-in rollout before any default switch.

Non-goals

• Do not replace the default base model or current classifier collection without reproducible evaluation evidence.
• Do not require one universal base model if separate reranker, embedding, and classifier bases are better for latency or quality.
• Do not optimize only for benchmark accuracy while ignoring latency, memory, or high-concurrency behavior.
• Do not require gated/private datasets or live network access in default CI.
• Do not silently change thresholds, config defaults, model aliases, or HF collection expectations without migration notes.

Acceptance criteria

• Current mmBERT/mmBERT-32K usage and current Hugging Face collection membership are inventoried and documented.
• A baseline report exists for current model quality, calibration, latency, throughput, memory, and long-context behavior.
• A candidate base-model shortlist is evaluated with a reproducible benchmark matrix.
• A next-generation model-family spec exists for reranker, embedding, and classifier artifacts, including naming, capabilities, runtime compatibility, and publication requirements.
• The classifier-family plan explicitly covers the current mom-multilingual-class tasks and describes migration/compatibility for existing merged and LoRA classifier assets.
• At least one recommended upgrade path is documented, including training/export/runtime compatibility requirements.
• Model lifecycle requirements from #2360 are satisfied for any proposed new base model or derived router model.
• Follow-up implementation issues are created for concrete model training, runtime integration, catalog updates, HF collection updates, docs, and default migration if the research finds viable candidates.

Validation

Initial research should at least run the repo-native routing report for affected surfaces:

make agent-report ENV=cpu CHANGED_FILES="src/semantic-router/pkg/classification src/semantic-router/pkg/config src/semantic-router/pkg/modelruntime src/semantic-router/pkg/modelinventory src/semantic-router/pkg/apiserver src/semantic-router/pkg/vectorstore src/semantic-router/pkg/cache src/training candle-binding onnx-binding nlp-binding ml-binding config deploy e2e website/docs"

Implementation follow-ups should add the smallest relevant gates, typically:

make agent-lint CHANGED_FILES="src/semantic-router/pkg/classification src/semantic-router/pkg/config src/semantic-router/pkg/modelruntime src/semantic-router/pkg/modelinventory src/semantic-router/pkg/apiserver src/semantic-router/pkg/vectorstore src/semantic-router/pkg/cache src/training candle-binding onnx-binding nlp-binding ml-binding config deploy e2e website/docs"
make agent-ci-gate CHANGED_FILES="src/semantic-router/pkg/classification src/semantic-router/pkg/config src/semantic-router/pkg/modelruntime src/semantic-router/pkg/modelinventory src/semantic-router/pkg/apiserver src/semantic-router/pkg/vectorstore src/semantic-router/pkg/cache src/training candle-binding onnx-binding nlp-binding ml-binding config deploy e2e website/docs"