research: evaluate base + multi-LoRA serving for router classifier assets

Summary

Evaluate whether router-owned classifier assets should support serving from one shared mmBERT-32K base model plus task-specific LoRA adapters instead of only serving multiple merged classifier models.

Parent roadmap: #2287 Parent model-family track: #2382 Related model lifecycle track: #2360 Related performance track: #2372 Current base model reference: https://huggingface.co/llm-semantic-router/mmbert-32k-yarn Current classifier collection: https://huggingface.co/collections/llm-semantic-router/mom-multilingual-class

Motivation

The current semantic-router defaults point to separate merged mmBERT-32K classifier artifacts for domain/intent, jailbreak/security, PII, fact-check, feedback, and modality routing. That makes deployment simple, but it duplicates the mmBERT backbone across multiple model directories and runtime classifier instances.

A base + multi-LoRA serving mode could reduce model package size, download time, cold-start cost, and resident memory by loading the shared base once and attaching small task adapters. This should be evaluated separately from a future shared-backbone multi-head classifier, because LoRA adapters attached inside attention/MLP layers may still require separate task forward passes even when base weights are shared.

Current implementation to compare against

• Default config uses multiple merged model paths under global.model_catalog.system and global.model_catalog.modules.
• Candle initializes task-specific mmBERT-32K classifier slots for intent, jailbreak, PII, fact-check, feedback, and modality.
• The unified LoRA batch path can run intent, PII, and security through a parallel engine, but it is still task-classifier based rather than one shared base forward with multiple heads.
• ONNX export currently favors merged classifier artifacts; dynamic base + adapter serving is expected to be Candle-first unless a separate ONNX strategy is designed.

Scope

• Inventory current merged and LoRA classifier artifacts for:
  • domain/intent;
  • jailbreak/security;
  • PII token classification;
  • fact-check;
  • feedback;
  • modality routing.
• Define a serving contract for one shared base model plus task adapters:
  • base model path and revision pin;
  • adapter path per task;
  • sequence-classification versus token-classification task type;
  • label/mapping files;
  • task thresholds and calibration metadata;
  • backend compatibility metadata.
• Evaluate runtime design options:
  • load one mmBERT-32K base and maintain an adapter registry;
  • load PEFT LoRA adapter weights and task heads in Candle;
  • select or batch adapters by task at inference time;
  • support sequence heads and token heads;
  • keep merged-model loading as fallback and compatibility path.
• Benchmark against the current multiple-merged-model path:
  • disk/artifact size;
  • image size and model download time;
  • cold-start and warmup time;
  • resident memory and peak memory;
  • p50/p95/p99 latency under single-task and multi-signal routing;
  • throughput under concurrent router requests;
  • CPU, CUDA, ROCm, and Candle behavior where available.
• Compare with a future shared-backbone multi-head classifier design and document which goal each design serves:
  • base + multi-LoRA: memory/package/cold-start optimization;
  • multi-head shared backbone: compute/latency optimization.

Non-goals

• Do not replace the current merged classifier defaults without benchmark evidence and migration notes.
• Do not assume base + multi-LoRA reduces latency unless measured; LoRA inside attention/MLP layers can require separate forward branches.
• Do not drop ONNX support. If ONNX cannot support dynamic adapters cleanly, keep merged ONNX artifacts as the ONNX path.
• Do not couple this work to a default model-family migration from #2382.
• Do not require live network access or large private datasets in default CI.

Affected code and surfaces

• config/config.yaml
• src/semantic-router/pkg/config/**
• src/semantic-router/pkg/classification/**
• src/semantic-router/pkg/modelruntime/**
• src/semantic-router/pkg/modelinventory/**
• candle-binding/src/ffi/**
• candle-binding/src/classifiers/lora/**
• candle-binding/src/model_architectures/lora/**
• candle-binding/src/model_architectures/traditional/modernbert.rs
• onnx-binding/**
• src/training/model_classifier/**
• scripts/export_classifiers_to_onnx.py
• website/docs/overview/mom-model-family.md

Acceptance criteria

• A short design note exists that explains whether base + multi-LoRA is worthwhile for router classifier serving and where it fits relative to merged models and multi-head classifiers.
• Current merged-model memory, package-size, cold-start, and latency baselines are recorded.
• A proposed config/catalog shape exists for base model plus adapter registry.
• Candle runtime gaps are identified for loading PEFT LoRA adapters, task heads, sequence classification, token classification, and adapter selection.
• ONNX compatibility is explicitly documented, including whether ONNX remains merged-artifact only.
• The migration plan preserves current *-merged classifier support and includes fallback/rollback behavior.
• Follow-up implementation issues are created if the design recommends building the feature.

Validation

Initial research/design should run the repo routing report for affected surfaces:

make agent-report ENV=cpu CHANGED_FILES="config src/semantic-router/pkg/config src/semantic-router/pkg/classification src/semantic-router/pkg/modelruntime src/semantic-router/pkg/modelinventory candle-binding onnx-binding src/training/model_classifier scripts/export_classifiers_to_onnx.py website/docs/overview/mom-model-family.md"

Implementation follow-ups should add the smallest relevant gates, typically:

make agent-lint CHANGED_FILES="src/semantic-router/pkg/config src/semantic-router/pkg/classification src/semantic-router/pkg/modelruntime src/semantic-router/pkg/modelinventory candle-binding onnx-binding src/training/model_classifier scripts/export_classifiers_to_onnx.py website/docs/overview/mom-model-family.md"
make agent-ci-gate CHANGED_FILES="src/semantic-router/pkg/config src/semantic-router/pkg/classification src/semantic-router/pkg/modelruntime src/semantic-router/pkg/modelinventory candle-binding onnx-binding src/training/model_classifier scripts/export_classifiers_to_onnx.py website/docs/overview/mom-model-family.md"