Repository metrics
- Stars
- (4,293 stars)
- PR merge metrics
- (PR metrics pending)
Description
Summary
Define a model onboarding procedure that turns a new candidate model into a validated router configuration update with benchmark evidence, model capability metadata, routing-policy impact, and monitoring expectations.
Parent roadmap: #2287
Related: #2333, #2334, #2358, #2359, #2360
Related code: src/semantic-router/pkg/modelselection, src/semantic-router/pkg/modelinventory, src/semantic-router/pkg/config/model_config_types.go, src/training, src/fleet-sim, deploy/recipes, website/docs
Motivation
The router config uses model endpoints, signal definitions, and routing decisions, but onboarding a new model still lacks a transparent procedure. A new model may change quality, latency, reasoning behavior, context limits, tool support, multimodal support, pricing, and fallback eligibility. Those properties should be evaluated and captured before the model is added to production recipes.
Scope
Define the onboarding flow for:
- benchmark and eval selection for latency, accuracy, reasoning, token usage, tool/multimodal capability, and cost;
- model capability inventory updates;
- recipe/model_config updates;
- routing-policy and threshold impact analysis;
- fleet-sim or load/capacity validation when relevant;
- monitoring and rollback expectations after activation.
Non-goals
- Do not require every model to pass the same benchmark set regardless of intended role.
- Do not silently update routing thresholds or recipes without eval evidence.
- Do not rely only on online learning to discover model regressions.
Acceptance criteria
- A contributor can follow a documented model onboarding checklist.
- Onboarding produces model metadata, eval evidence, config changes, and validation receipts.
- The process identifies whether the model is usable for routing, fallback, tools, multimodal, or special recipes.
- Existing recipes can opt into the new model through explicit config changes.
Validation
make agent-report ENV=cpu CHANGED_FILES="src/semantic-router/pkg/modelselection src/semantic-router/pkg/modelinventory src/semantic-router/pkg/config src/training src/fleet-sim deploy/recipes website/docs"
make agent-lint CHANGED_FILES="src/semantic-router/pkg/modelselection src/semantic-router/pkg/modelinventory src/semantic-router/pkg/config src/training src/fleet-sim deploy/recipes website/docs"
make agent-ci-gate CHANGED_FILES="src/semantic-router/pkg/modelselection src/semantic-router/pkg/modelinventory src/semantic-router/pkg/config src/training src/fleet-sim deploy/recipes website/docs"