vllm-project/semantic-router

[Router] Text-bearing image content can over-fire embedding meta-rules without a stage-2 text-modality check

Open

#2,057 opened on Jun 5, 2026

View on GitHub
 (1 comment) (0 reactions) (0 assignees)Go (699 forks)github user discovery
area/corehelp wantedroadmapsignal-decision-engine

Repository metrics

Stars
 (4,293 stars)
PR merge metrics
 (PR metrics pending)

Description

Summary

When the embedding signal routes image-modality content, text-bearing images (whiteboards, scanned documents, screenshots) tend to over-fire the cheap image-embedding meta-rules, while their actual sensitivity (for example PII in the rendered text) is not reliably captured by image-embedding similarity alone. Catching it reliably needs a text-modality check at stage 2 (OCR feeding text classification) on the flagged candidates.

Why it matters

A privacy/routing layer that decides image-modality sensitivity from image-embedding similarity alone can mis-route text-bearing-but-sensitive images, because the signal that matters (the rendered text) is not in the image-embedding space.

Evidence

In local experiments, a staged approach (cheap image-embedding meta-rules flag candidates, then a text-modality resolver of OCR plus text classification decides sensitivity on the flagged set, with sibling meta-rules tuned per content band such as digital screens, scanned documents, and publication layouts) took safe-side errors from 23 to 4 across four iterative changes on a stratified ~120-fixture held-out corpus (DocVQA val plus public HuggingFace UI-screenshot datasets). Reproducible receipts can be shared with maintainers.

Scope

This is future-facing robustness for the image-routing layer, not a v0.3 blocker. It needs a design discussion on where the stage-2 text-modality check lives and how it composes with the existing signal pipeline before it is more than a flag.

H2 execution scope

Use this issue for the text-bearing image stage-2 check inside the multimodal/image-routing robustness parent #2347. The stage-2 check should prevent rendered text images from over-firing image embedding meta-rules when a text-modality or OCR-style signal is needed.

Relevant code surfaces: src/semantic-router/pkg/classification, multimodal embedding dispatch, image API handling, and #2347 diagnostics.

Acceptance criteria

  • Text-bearing image examples reproduce the over-fire behavior.
  • A stage-2 text-modality/OCR-style design is documented or implemented.
  • The router can distinguish image semantic similarity from rendered-text sensitivity.
  • Replay/diagnostics show when the stage-2 check was applied or unavailable.

Validation

make agent-report ENV=cpu CHANGED_FILES="src/semantic-router/pkg/classification src/semantic-router/pkg/config src/semantic-router/pkg/apiserver candle-binding nlp-binding website/docs"
make agent-lint CHANGED_FILES="src/semantic-router/pkg/classification src/semantic-router/pkg/config src/semantic-router/pkg/apiserver candle-binding nlp-binding website/docs"
make agent-ci-gate CHANGED_FILES="src/semantic-router/pkg/classification src/semantic-router/pkg/config src/semantic-router/pkg/apiserver candle-binding nlp-binding website/docs"

Contributor guide