vllm-project/semantic-router

feat: add a decision-level routing evaluation harness

Open

#2,333 opened on Jul 5, 2026

View on GitHub
 (1 comment) (0 reactions) (1 assignee)Go (699 forks)github user discovery
area/agentarea/model-selectionarea/testingenhancementevaluationhelp wantedpriority/P1roadmap

Repository metrics

Stars
 (4,293 stars)
PR merge metrics
 (PR metrics pending)

Description

Summary

Add a decision-level evaluation harness that measures whether the router selected the correct route or model for a dataset of requests. This fills the gap between signal-level evaluation and end-to-end system quality evaluation.

Parent roadmap: #2287 Related: #2238, #2244, #2330, #2331

Motivation

Production adoption and Router Learning both need a guardrail that answers a specific question: did the routing decision itself do the right thing?

Signal-level eval can say whether an extractor fired correctly. E2E eval can say whether the final model answer was good. Neither isolates the routing decision across configured signals, projections, decisions, recipes, and model selection. Without this harness, adaptive thresholds, agentic config tuning, learning algorithms, and recipe changes are hard to review safely.

Scope

The harness should:

  • load a routing config or recipe;
  • load a dataset with request payloads and expected route/model outcomes;
  • run routing decisions without requiring upstream model completion when possible;
  • compare expected vs actual route/model/decision;
  • report regressions across config changes;
  • include coverage for complexity, domain, intent, safety, and custom signal outputs;
  • produce stable JSON plus concise human-readable summaries for PR and release review.

Non-goals

  • Do not train or tune signal models in this issue.
  • Do not implement Router Learning algorithms in this issue.
  • Do not replace E2E answer-quality benchmarks.

Acceptance criteria

  • A maintained decision-level eval dataset format is documented.
  • The harness can grade route correctness for at least one recipe.
  • Reports include expected route/model, actual route/model, selected decision, selected recipe when available, and signal/projection evidence.
  • CI/release profiles can run the harness deterministically.
  • The output can be used as a guardrail for adaptive thresholds and agentic config tuning.

Validation

make agent-report ENV=cpu CHANGED_FILES="bench src/semantic-router/pkg/classification src/semantic-router/pkg/decision src/semantic-router/pkg/selection config deploy/recipes website/docs"
make agent-lint CHANGED_FILES="bench src/semantic-router/pkg/classification src/semantic-router/pkg/decision src/semantic-router/pkg/selection config deploy/recipes website/docs"
make agent-ci-gate CHANGED_FILES="bench src/semantic-router/pkg/classification src/semantic-router/pkg/decision src/semantic-router/pkg/selection config deploy/recipes website/docs"

Contributor guide