vllm-project/semantic-router

[Router] image-routing.yaml thresholds need recalibration after the encoder-correctness fixes (#1927, #1928, #1943)

Open

#2,165 opened on Jun 12, 2026

View on GitHub
 (2 comments) (0 reactions) (1 assignee)Go (699 forks)github user discovery
area/corebughelp wantedpriority/P2roadmapsignal-decision-engine

Repository metrics

Stars
 (4,293 stars)
PR merge metrics
 (PR metrics pending)

Description

The opt-in image-routing embedding pack (config/signal/embedding/image-routing.yaml, added in #1896) ships thresholds in the 0.04-0.17 range that were calibrated against the encoder's pre-fix behavior. After the pooling-head (#1927), normalization (#1928), and PIL-equivalent resize (#1943) fixes landed, runtime cosine scores against multi-modal-embed-small sit in a different band, so the shipped thresholds no longer reflect measured separations.

Recalibration is measurement-gated: it needs a runtime cosine re-measure of the 28-candidate set against multi-modal-embed-small on a live deployment (the calibration corpus and method are in the pack's tutorial section). Until that lands, any docs change touching these numbers should hold rather than republish the stale range.

Acceptance:

  • re-measured per-rule cosine bands against the current encoder, corpus committed alongside (probe-dir convention)
  • updated thresholds in image-routing.yaml + tutorial text
  • a calibration-provenance note stating which encoder build produced the numbers

H2 execution scope

Recalibrate image-routing thresholds after encoder correctness fixes and record the measurement provenance. This is a child of #2347 and should not be a blind config tweak.

Relevant code surfaces: image routing configs, src/semantic-router/pkg/classification, multimodal embedding bindings, and docs/bench artifacts.

Acceptance criteria

  • Threshold changes are backed by a documented calibration dataset and metric report.
  • False-positive/false-negative behavior is compared before and after recalibration.
  • Config changes identify encoder/model version assumptions.
  • CI or manual receipt policy covers the relevant multimodal path.

Validation

make agent-report ENV=cpu CHANGED_FILES="src/semantic-router/pkg/classification src/semantic-router/pkg/config candle-binding nlp-binding config website/docs bench"
make agent-lint CHANGED_FILES="src/semantic-router/pkg/classification src/semantic-router/pkg/config candle-binding nlp-binding config website/docs bench"
make agent-ci-gate CHANGED_FILES="src/semantic-router/pkg/classification src/semantic-router/pkg/config candle-binding nlp-binding config website/docs bench"

Contributor guide