vllm-project/semantic-router

feat: harden multimodal and image-routing signal robustness

Open

#2,347 opened on Jul 5, 2026

View on GitHub
 (0 comments) (0 reactions) (1 assignee)Go (699 forks)github user discovery
area/corearea/observabilityarea/testingenhancementhelp wantedpriority/P1roadmapsignal-decision-engine

Repository metrics

Stars
 (4,293 stars)
PR merge metrics
 (PR metrics pending)

Description

Summary

Harden multimodal and image-routing signal behavior so image-modality routing can be evaluated, calibrated, discovered, and tested reliably across router APIs and embedding backends.

Parent roadmap: #2287 Related signal roadmap: #2330, #2335, #2333

Motivation

Image-routing is becoming a broader signal surface: request-time routing, HTTP classify/similarity endpoints, model discovery, threshold packs, and CI coverage all depend on multimodal embeddings behaving consistently. Several open issues show that the current surface has correctness and operational gaps:

  • text-bearing images can over-fire image embedding meta-rules without a stage-2 text-modality check;
  • image-routing thresholds need recalibration after encoder correctness fixes;
  • ONNX and Candle image preprocessing may diverge;
  • HTTP APIs do not yet consistently support image inputs across classify/batch/similarity;
  • clients cannot discover whether multimodal embedding support is loaded;
  • model-gated multimodal tests do not run in CI.

This parent issue groups that work so it is visible in the H2 roadmap instead of remaining scattered.

Child issues

  • #2057: Text-bearing image content needs a stage-2 text-modality check.
  • #2165: Recalibrate image-routing thresholds after encoder correctness fixes.
  • #2166: Resolve ONNX/Candle image preprocessing parity.
  • #2317: Extend HTTP API image input to classify/batch and similarity endpoints.
  • #2318: Expose multimodal model capability discovery.
  • #2319: Run or explicitly gate multimodal model tests in CI.

Scope

This track should define and validate:

  • stage-2 OCR/text-modality checks for text-bearing images;
  • measured threshold calibration with committed provenance;
  • binding parity or fail-closed behavior for unsupported multimodal paths;
  • API capability discovery for multimodal models;
  • endpoint behavior for image classification and similarity;
  • CI or manual receipt policy for model-dependent multimodal tests;
  • replay/diagnostic output for multimodal signal decisions.

Non-goals

  • Do not make multimodal support mandatory for all deployments.
  • Do not silently mix incompatible embedding spaces.
  • Do not treat image embedding similarity as sufficient for rendered-text sensitivity.
  • Do not broaden the API surface without validation and capability discovery.

Acceptance criteria

  • Multimodal/image-routing child issues have explicit owners or sequencing.
  • Image-routing thresholds have measurement provenance.
  • Text-bearing image routing has a stage-2 design or implementation path.
  • Multimodal support is discoverable by clients.
  • CI or release validation covers at least one meaningful multimodal encode path, or the manual receipt policy is documented.
  • The H2 roadmap links this parent issue as the multimodal signal robustness track.

Validation

make agent-report ENV=cpu CHANGED_FILES="src/semantic-router/pkg/classification src/semantic-router/pkg/config src/semantic-router/pkg/apiserver candle-binding onnx-binding config website/docs bench .github"
make agent-lint CHANGED_FILES="src/semantic-router/pkg/classification src/semantic-router/pkg/config src/semantic-router/pkg/apiserver candle-binding onnx-binding config website/docs bench .github"
make agent-ci-gate CHANGED_FILES="src/semantic-router/pkg/classification src/semantic-router/pkg/config src/semantic-router/pkg/apiserver candle-binding onnx-binding config website/docs bench .github"

Contributor guide