vllm-project/semantic-router

feat: add adaptive thresholding and policy calibration

Open

#2,341 opened on Jul 5, 2026

View on GitHub
 (5 comments) (0 reactions) (1 assignee)Go (699 forks)github user discovery
area/model-selectionarea/researchenhancementhelp wantedpriority/P2roadmapsignal-decision-engine

Repository metrics

Stars
 (4,293 stars)
PR merge metrics
 (PR metrics pending)

Description

Summary

Add adaptive thresholding and policy calibration so routing thresholds can be tuned from evaluation, feedback, and confidence evidence instead of remaining fixed static constants.

Parent roadmap: #2287 Related: #2322, #2330, #2333, #2340

Motivation

Static thresholds such as fixed similarity cutoffs can be brittle across domains, tenants, datasets, and recipes. Adaptive calibration can improve route quality, but it needs evaluation guardrails to avoid overfitting or unsafe drift.

Scope

This issue should define and implement calibrated threshold workflows for:

  • signal confidence thresholds;
  • projection score thresholds;
  • decision guard thresholds;
  • ambiguity and fallback thresholds;
  • per-recipe calibration profiles.

Calibration should be driven by offline eval/replay evidence and should produce explicit candidate config changes or snapshot metadata.

Non-goals

  • Do not silently mutate thresholds on the request path.
  • Do not introduce adaptive behavior without a decision-level eval guardrail.
  • Do not merge this with general Router Learning algorithms; this is policy calibration.

Acceptance criteria

  • Threshold calibration inputs and outputs are documented.
  • At least one threshold family can be calibrated from an eval dataset.
  • Calibrated thresholds are represented as explicit config or snapshot changes.
  • Reports show quality, false-positive, false-negative, cost, and regression impacts.

Contributor guide