vllm-project/semantic-router

[Router] Extend HTTP API image input to classify/batch and the similarity endpoints (deferred from #2104)

Open

#2,317 opened on Jul 3, 2026

View on GitHub
 (2 comments) (0 reactions) (1 assignee)Go (699 forks)github user discovery
area/corehelp wantedroadmapsignal-decision-engine

Repository metrics

Stars
 (4,293 stars)
PR merge metrics
 (PR metrics pending)

Description

What

#2104 (under review) delivers image input for /api/v1/classify/intent, /api/v1/eval, and /api/v1/embeddings - the first slices of #1911's staged delivery. Its body explicitly defers the remaining endpoints; this issue tracks them so the deferral has a home when #2104's "Closes #1911" fires at merge:

  • POST /api/v1/classify/batch (BatchClassificationRequest - texts only today)
  • POST /api/v1/similarity (SimilarityRequest)
  • POST /api/v1/similarity/batch (BatchSimilarityRequest)

Design notes carried from the #2104 review

  • Reuse pkg/utils/imageurl as the single input gate, including whatever payload-extraction consolidation lands from that review's findings.
  • The similarity endpoints are where the embedding-space question bites hardest: text routes to qwen3/gemma/mmbert while images route to the multimodal projection, and cosine across spaces is meaningless. MultiModalEncodeText exists in candle-binding but is unreachable from these endpoints - image-vs-text similarity likely requires a model: "multimodal" option routing BOTH sides into the shared space. See the design question on #2104.
  • Dimension semantics: the multimodal model's native dimension is 384 while the API default is 768 - the over-dimension contract (#2104 review) should be settled before these endpoints wire in.
  • Stage-2 consideration: text-bearing images (screenshots, scanned documents) carry their routing signal in the rendered text, not the image embedding. #2057 tracks the stage-2 text-modality check for the ExtProc path; every HTTP endpoint added here widens the same surface, so any stage-2 design should cover both entry paths.

Why now

#1911 was stale-closed while #2104 was in flight (since reopened). When #2104 merges it will close #1911, and without this issue the deferred endpoints lose tracking - staged-delivery issues are exactly what stale bots eat.

H2 execution scope

Extend image input handling consistently across classify, batch classify, and similarity endpoints as part of #2347 multimodal robustness.

Relevant code surfaces: src/semantic-router/pkg/apiserver, src/semantic-router/pkg/classification, multimodal embedding dispatch, config validation, and API docs.

Acceptance criteria

  • Classify, batch classify, and similarity endpoints have documented image input shapes.
  • Unsupported model/runtime paths return clear capability errors.
  • Tests cover successful image input and unsupported/fallback behavior.
  • API docs and capability discovery are updated together.

Contributor guide