vllm-project/semantic-router

[Bindings] onnx-binding image preprocessing diverges from candle-binding's PIL-equivalent resize

Open

#2,166 opened on Jun 12, 2026

View on GitHub
 (1 comment) (0 reactions) (1 assignee)Go (699 forks)github user discovery
area/corebughelp wantedpriority/P2roadmapsignal-decision-engine

Repository metrics

Stars
 (4,293 stars)
PR merge metrics
 (PR metrics pending)

Description

#1943 moved candle-binding image preprocessing to a Catmull-Rom support-weighted resize that tracks PIL bicubic+antialias to ~1e-4 cosine. onnx-binding's image path still uses its previous resize, so the two bindings now produce embeddings from different pixel pipelines for the same input image. Surfaced while reviewing the #1943 figure record.

Two candidate resolutions, maintainer call needed:

  1. Port the PIL-equivalent resize to onnx-binding (parity, but more code in a binding whose multimodal surface is still settling).
  2. Fail closed: have onnx-binding declare MultimodalEmbedding: false until its preprocessing reaches parity, so a mixed deployment cannot silently mix pixel pipelines.

Either way, a parity probe (same fixtures, both bindings, cosine matrix) should be the first step; no committed measurement currently exists for the onnx path.

H2 execution scope

Resolve ONNX/Candle image preprocessing parity or make unsupported divergent paths fail closed. This is a correctness dependency for #2347 and image-routing threshold calibration.

Relevant code surfaces: candle-binding, onnx-binding if present, image embedding dispatch in src/semantic-router/pkg/classification, and multimodal tests.

Acceptance criteria

  • ONNX and Candle preprocessing differences are measured on shared fixtures.
  • The implementation either matches preprocessing behavior or explicitly blocks divergent unsupported paths.
  • Parity tests cover resize/crop/normalization behavior that affects embeddings.
  • Downstream threshold calibration notes the fixed preprocessing path.

Validation

make agent-report ENV=cpu CHANGED_FILES="candle-binding onnx-binding src/semantic-router/pkg/classification src/semantic-router/pkg/config website/docs"
make agent-lint CHANGED_FILES="candle-binding onnx-binding src/semantic-router/pkg/classification src/semantic-router/pkg/config website/docs"
make agent-ci-gate CHANGED_FILES="candle-binding onnx-binding src/semantic-router/pkg/classification src/semantic-router/pkg/config website/docs"

Contributor guide