vllm-project/semantic-router

feature: add an optional generative-LLM extractor for complexity signals

Open

#2,154 opened on Jun 11, 2026

View on GitHub
 (3 comments) (1 reaction) (0 assignees)Go (699 forks)github user discovery
area/researchhelp wantedroadmapsignal-decision-engine

Repository metrics

Stars
 (4,293 stars)
PR merge metrics
 (PR metrics pending)

Description

Problem

Embedding/prototype similarity can be weak when the query and reviewed easy/hard examples differ in language, style, or domain. Some deployments want a small generative judge to emit a typed complexity verdict while keeping the existing complexity signal and decision rules.

Audited upstream baseline (2026-07-15)

Current main computes complexity from prototype similarity over text candidates and optional image candidates. The embedding provider can be local or an OpenAI-compatible remote embedding endpoint; the implementation is not limited to local mmBERT. ComplexityModelConfig currently exposes prototype-scoring controls, and there is no generative-LLM complexity extractor.

The existing VLLMJailbreakInference is useful prior art for authenticated OpenAI-compatible inference, but its guard-specific parsing and lifecycle should not be copied as the public complexity contract.

Scope

  1. Define an optional typed complexity extractor behind the canonical complexity model module.
  2. Support a reviewed local or OpenAI-compatible generative endpoint through the shared model-runtime/provider boundary.
  3. Parse a constrained result into the existing easy, medium, and hard rule outputs with explicit confidence/provenance.
  4. Define timeout, invalid-output, and provider-error behavior. The safe default is the existing prototype path or an explicit unknown result, not a silently fabricated verdict.
  5. Keep prototype scoring as the default and support shadow evaluation before activation.
  6. Add multilingual and domain-shift evaluation against prototype and simple heuristic baselines.

Boundaries

  • Do not add a legacy method-keyed external_models contract if the canonical model catalog/runtime can own the dependency.
  • Do not send prompts to an external judge without explicit operator configuration and privacy documentation.
  • Do not promise latency or accuracy from a single local prototype measurement.
  • Do not change existing rule_name:easy|medium|hard decision references.

Acceptance criteria

  • The LLM extractor is optional, typed, and discoverable in diagnostics.
  • Invalid or ambiguous output cannot become a confident route.
  • Timeout/error/fallback behavior is deterministic and tested.
  • Credentials are resolved through the supported provider boundary and are never logged or stored.
  • Evaluation reports quality, latency, cost, and multilingual behavior on a representative dataset.
  • Existing prototype-only configurations behave unchanged.

Relevant surfaces

src/semantic-router/pkg/classification, src/semantic-router/pkg/config/model_config_types.go, src/semantic-router/pkg/modelruntime, decision evaluation (#2333), docs, and tests.

Contributor guide