feat: harden multimodal and image-routing signal robustness
#2,347 opened on Jul 5, 2026
Repository metrics
- Stars
- (4,293 stars)
- PR merge metrics
- (PR metrics pending)
Description
Summary
Harden multimodal and image-routing signal behavior so image-modality routing can be evaluated, calibrated, discovered, and tested reliably across router APIs and embedding backends.
Parent roadmap: #2287 Related signal roadmap: #2330, #2335, #2333
Motivation
Image-routing is becoming a broader signal surface: request-time routing, HTTP classify/similarity endpoints, model discovery, threshold packs, and CI coverage all depend on multimodal embeddings behaving consistently. Several open issues show that the current surface has correctness and operational gaps:
- text-bearing images can over-fire image embedding meta-rules without a stage-2 text-modality check;
- image-routing thresholds need recalibration after encoder correctness fixes;
- ONNX and Candle image preprocessing may diverge;
- HTTP APIs do not yet consistently support image inputs across classify/batch/similarity;
- clients cannot discover whether multimodal embedding support is loaded;
- model-gated multimodal tests do not run in CI.
This parent issue groups that work so it is visible in the H2 roadmap instead of remaining scattered.
Child issues
- #2057: Text-bearing image content needs a stage-2 text-modality check.
- #2165: Recalibrate image-routing thresholds after encoder correctness fixes.
- #2166: Resolve ONNX/Candle image preprocessing parity.
- #2317: Extend HTTP API image input to classify/batch and similarity endpoints.
- #2318: Expose multimodal model capability discovery.
- #2319: Run or explicitly gate multimodal model tests in CI.
Scope
This track should define and validate:
- stage-2 OCR/text-modality checks for text-bearing images;
- measured threshold calibration with committed provenance;
- binding parity or fail-closed behavior for unsupported multimodal paths;
- API capability discovery for multimodal models;
- endpoint behavior for image classification and similarity;
- CI or manual receipt policy for model-dependent multimodal tests;
- replay/diagnostic output for multimodal signal decisions.
Non-goals
- Do not make multimodal support mandatory for all deployments.
- Do not silently mix incompatible embedding spaces.
- Do not treat image embedding similarity as sufficient for rendered-text sensitivity.
- Do not broaden the API surface without validation and capability discovery.
Acceptance criteria
- Multimodal/image-routing child issues have explicit owners or sequencing.
- Image-routing thresholds have measurement provenance.
- Text-bearing image routing has a stage-2 design or implementation path.
- Multimodal support is discoverable by clients.
- CI or release validation covers at least one meaningful multimodal encode path, or the manual receipt policy is documented.
- The H2 roadmap links this parent issue as the multimodal signal robustness track.
Validation
make agent-report ENV=cpu CHANGED_FILES="src/semantic-router/pkg/classification src/semantic-router/pkg/config src/semantic-router/pkg/apiserver candle-binding onnx-binding config website/docs bench .github"
make agent-lint CHANGED_FILES="src/semantic-router/pkg/classification src/semantic-router/pkg/config src/semantic-router/pkg/apiserver candle-binding onnx-binding config website/docs bench .github"
make agent-ci-gate CHANGED_FILES="src/semantic-router/pkg/classification src/semantic-router/pkg/config src/semantic-router/pkg/apiserver candle-binding onnx-binding config website/docs bench .github"