[Router] Text-bearing image content can over-fire embedding meta-rules without a stage-2 text-modality check
#2,057 opened on Jun 5, 2026
Repository metrics
- Stars
- (4,293 stars)
- PR merge metrics
- (PR metrics pending)
Description
Summary
When the embedding signal routes image-modality content, text-bearing images (whiteboards, scanned documents, screenshots) tend to over-fire the cheap image-embedding meta-rules, while their actual sensitivity (for example PII in the rendered text) is not reliably captured by image-embedding similarity alone. Catching it reliably needs a text-modality check at stage 2 (OCR feeding text classification) on the flagged candidates.
Why it matters
A privacy/routing layer that decides image-modality sensitivity from image-embedding similarity alone can mis-route text-bearing-but-sensitive images, because the signal that matters (the rendered text) is not in the image-embedding space.
Evidence
In local experiments, a staged approach (cheap image-embedding meta-rules flag candidates, then a text-modality resolver of OCR plus text classification decides sensitivity on the flagged set, with sibling meta-rules tuned per content band such as digital screens, scanned documents, and publication layouts) took safe-side errors from 23 to 4 across four iterative changes on a stratified ~120-fixture held-out corpus (DocVQA val plus public HuggingFace UI-screenshot datasets). Reproducible receipts can be shared with maintainers.
Scope
This is future-facing robustness for the image-routing layer, not a v0.3 blocker. It needs a design discussion on where the stage-2 text-modality check lives and how it composes with the existing signal pipeline before it is more than a flag.
H2 execution scope
Use this issue for the text-bearing image stage-2 check inside the multimodal/image-routing robustness parent #2347. The stage-2 check should prevent rendered text images from over-firing image embedding meta-rules when a text-modality or OCR-style signal is needed.
Relevant code surfaces: src/semantic-router/pkg/classification, multimodal embedding dispatch, image API handling, and #2347 diagnostics.
Acceptance criteria
- Text-bearing image examples reproduce the over-fire behavior.
- A stage-2 text-modality/OCR-style design is documented or implemented.
- The router can distinguish image semantic similarity from rendered-text sensitivity.
- Replay/diagnostics show when the stage-2 check was applied or unavailable.
Validation
make agent-report ENV=cpu CHANGED_FILES="src/semantic-router/pkg/classification src/semantic-router/pkg/config src/semantic-router/pkg/apiserver candle-binding nlp-binding website/docs"
make agent-lint CHANGED_FILES="src/semantic-router/pkg/classification src/semantic-router/pkg/config src/semantic-router/pkg/apiserver candle-binding nlp-binding website/docs"
make agent-ci-gate CHANGED_FILES="src/semantic-router/pkg/classification src/semantic-router/pkg/config src/semantic-router/pkg/apiserver candle-binding nlp-binding website/docs"