Modularize native router model bindings and runtime capability contracts

Summary

The native router model path has grown into several backend-specific binding surfaces:

• Candle is the default CGO/Rust path for BERT, ModernBERT, mmBERT, mmBERT-32K classifiers, Qwen3/Gemma/mmBERT/multimodal embeddings, Qwen3 multi-LoRA classification, Qwen3Guard safety classification, similarity, MLP selector, and modality routing.
• ONNX Runtime has a separate Rust/Go binding for mmBERT embedding, 2D Matryoshka, and named mmBERT sequence/token classifiers, with CPU, ROCm, and CUDA provider strategies.
• OpenVINO has a separate C++/Go binding for ModernBERT sequence classification, token classification, and embeddings.
• Remote embedding providers and future native backends need to fit the same router-facing model lifecycle, capability, diagnostics, and validation story.

This issue tracks a roadmap-level refactor to make native model serving extensible through binding-neutral contracts instead of scattered build tags, stringly-typed model kinds, duplicated FFI discriminants, and backend-specific singleton state.

Current Problems

• The Go router has a useful NativeBackendCapabilities seam, but the real runtime calls still reach directly into candle-binding, onnx-binding, or openvino-binding helpers from classification, embedding, vector store, modelruntime, and API model-info paths.
• Candle exposes many specialized init/classify/embed functions through one large Go CGO facade, while Rust-side model state is mostly role-specific global OnceLock storage.
• ONNX uses a more extensible named-classifier map for sequence/token classifiers, but unified/LoRA compatibility paths are currently stubs and capability parity is not represented as a first-class contract.
• OpenVINO has its own build-tag and helper path, with similar classifier/embedding concepts but no shared backend adapter interface.
• Model taxonomy is duplicated across Rust enums, C ABI integer discriminants, Go string model types, config defaults, validators, registry metadata, and API model-info metadata. Some comments and constants have already drifted from implemented behavior.
• The current taxonomy mixes several axes: backend, model family, task/capability, artifact format, LoRA form, modality, provider, batching, and optimization features.
• ModelFactory is a promising abstraction but currently mixes classification and embedding concerns and has unimplemented Traditional/LoRA reference creation paths.

Goals

• Define a binding-neutral native runtime contract for router-owned model assets.
• Make Candle, ONNX Runtime, OpenVINO, and remote providers adapters behind one router-facing lifecycle and capability interface.
• Separate taxonomy axes explicitly:
  • backend: candle, onnxruntime, openvino, remote, future backends;
  • capability: embedding, sequence classification, token classification, multimodal embedding, modality routing, reranking, guard, generative classification, LoRA adapter serving, MLP selector;
  • family: BERT, ModernBERT, mmBERT, Qwen3, Gemma, MiniLM, Qwen3Guard, future families;
  • artifact: full model, merged LoRA, adapter, ONNX IR, OpenVINO IR, quantized asset;
  • modality: text, image, audio, multimodal;
  • runtime features: batching, 2D Matryoshka, layer early exit, provider selection, explicit reset, concurrent load, hot-swap support.
• Keep the router request path backend-neutral: callers ask for a capability and model ref, not a concrete binding package.
• Make capability discovery accurate enough for config validation, dashboard/API model info, E2E selection, and deployment docs.
• Preserve existing default behavior and config compatibility while enabling incremental migration.

Non-Goals

• Do not rewrite all inference kernels in one pass.
• Do not remove Candle as the default backend.
• Do not require every backend to support every capability immediately.
• Do not change default model choices as part of the first contract refactor.

Proposed Design Direction

1. Native Runtime Contract

Introduce a router-owned native runtime layer, for example under src/semantic-router/pkg/modelruntime/native or an equivalent package, with concepts like:

• BackendAdapter: reports backend name, build/runtime provider info, and supported capabilities.
• ModelHandle: stable handle for loaded model assets, keyed by model ref, capability, family, artifact, backend, and optional version/provider.
• LoadRequest: model ref, resolved path, capability, family, artifact format, device/provider policy, batching hints, target dimensions/layers, and labels/mappings.
• InferenceRequest: typed requests for embedding, sequence classification, token classification, multimodal embedding, guard, reranking, and generative classification.
• CapabilitySet: structured capability flags and limits, not one-off booleans hidden in binding code.
• ModelInfo: one normalized runtime model-info shape used by /models, embedding model discovery, startup status, dashboard, and diagnostics.

2. Taxonomy and ABI Lockdown

Create one canonical native model taxonomy and keep binding-specific representations generated from it or covered by tests.

Acceptance should include:

• Go constants for all supported embedding and classifier family values, not only a subset.
• Rust/C/Go discriminant tests for FFI enum-like fields such as embedding model_type.
• Struct layout tests for FFI payloads where Go typedefs must match Rust/C++ repr(C) structures.
• Validation tests that reject unsupported capability/backend combinations before first request-time inference.

3. Candle Adapter Cleanup

Migrate Candle behind the adapter interface first because it is the default and broadest backend.

Recommended steps:

• Split the large Go CGO facade by capability while keeping public compatibility wrappers during migration.
• Replace role-specific singleton sprawl with a central model registry/state manager, or explicitly document which Candle capabilities remain process-singleton and why.
• Decide whether ModelFactory is the production extensibility seam. If yes, finish Traditional/LoRA reference handling and separate classification, embedding, multimodal, and generative concerns. If no, move it out of the request-critical path and make the new adapter registry the authoritative seam.
• Route existing BERT/ModernBERT/mmBERT classifier, Qwen3/Gemma/mmBERT embedding, Qwen3 multi-LoRA, and Qwen3Guard paths through the same capability contract.

4. ONNX Runtime and OpenVINO Adapters

Bring ONNX Runtime and OpenVINO into the same contract without pretending feature parity exists.

• ONNX should report mmBERT embedding, 2D Matryoshka, provider selection, and named mmBERT sequence/token classifier capabilities explicitly.
• ONNX unified/LoRA stubs should be represented as unsupported capabilities at discovery/validation time rather than surprising request-path failures.
• OpenVINO should report ModernBERT classifier and embedding support, device/provider policy, tokenizer/model layout expectations, and unsupported features such as missing Matryoshka truncation where applicable.
• Deployment docs for AMD, NVIDIA, CPU, and Intel paths should consume the same capability vocabulary.

5. Router Integration

Refactor call sites incrementally so router code depends on the native runtime contract:

• classification backend selection;
• embedding classifier initialization and request embedding;
• semantic cache/vector store/memory embedding warmup;
• modality classifier init and classification;
• model info and capability discovery APIs;
• startup status and diagnostics;
• CLI/Kubernetes config validation.

A new backend or model family should require a new adapter registration and model catalog entry, not edits across every caller.

6. Validation and E2E Matrix

Add focused tests before broad rewrites:

• ABI drift tests for Go/Rust/C++ binding structs and discriminants.
• Unit tests for capability discovery and unsupported-capability errors.
• Config validation tests for backend/capability/family/artifact combinations.
• E2E or model-gated receipts for Candle default paths.
• Backend-specific receipts for ONNX Runtime and OpenVINO where models or provider libraries are optional.

Acceptance Criteria

• There is a documented native runtime contract covering Candle, ONNX Runtime, OpenVINO, remote embeddings, and future backends.
• The contract separates backend, capability, family, artifact, modality, and runtime feature axes.
• Existing Candle capabilities are reachable through the new adapter layer while compatibility wrappers continue to work.
• ONNX Runtime and OpenVINO capabilities are discoverable through the same API shape, including unsupported capability reasons.
• /models or an equivalent model-info endpoint reports backend, capability, family, artifact, modality, dimensions/layers, provider/device, loaded state, and registry metadata consistently.
• Go/Rust/C++ ABI drift is covered by tests or generated declarations.
• Config validators catch unsupported backend/model/capability combinations before serving traffic.
• At least one new backend or model family can be added by implementing an adapter and registry entry without touching classification, embedding, vector store, modelruntime, and API call sites individually.
• Legacy configs and current default Candle behavior remain compatible during migration.

Related Issues

• #2287: H2 roadmap index.
• #2394: Base + multi-LoRA serving for router classifier assets.
• #2395: MIGraphX-first ONNX Runtime provider strategy for AMD router signal models.
• #2375: Deep router code-quality, security, resource-safety, and API-boundary audit.
• #2318: Multimodal model capability discovery.
• #1510: Large-scale performance regression coverage across Candle/ONNX, NVIDIA, and AMD.

Suggested Workstream Placement

This belongs under the H2 roadmap workstream for signal, decision, provider capability, and model-quality foundation, with cross-links to backend-aware routing/performance and code-quality hardening.