feature: add an optional generative-LLM extractor for complexity signals
#2,154 opened on Jun 11, 2026
Repository metrics
- Stars
- (4,293 stars)
- PR merge metrics
- (PR metrics pending)
Description
Problem
Embedding/prototype similarity can be weak when the query and reviewed easy/hard examples differ in language, style, or domain. Some deployments want a small generative judge to emit a typed complexity verdict while keeping the existing complexity signal and decision rules.
Audited upstream baseline (2026-07-15)
Current main computes complexity from prototype similarity over text candidates and optional image candidates. The embedding provider can be local or an OpenAI-compatible remote embedding endpoint; the implementation is not limited to local mmBERT. ComplexityModelConfig currently exposes prototype-scoring controls, and there is no generative-LLM complexity extractor.
The existing VLLMJailbreakInference is useful prior art for authenticated OpenAI-compatible inference, but its guard-specific parsing and lifecycle should not be copied as the public complexity contract.
Scope
- Define an optional typed complexity extractor behind the canonical complexity model module.
- Support a reviewed local or OpenAI-compatible generative endpoint through the shared model-runtime/provider boundary.
- Parse a constrained result into the existing
easy,medium, andhardrule outputs with explicit confidence/provenance. - Define timeout, invalid-output, and provider-error behavior. The safe default is the existing prototype path or an explicit unknown result, not a silently fabricated verdict.
- Keep prototype scoring as the default and support shadow evaluation before activation.
- Add multilingual and domain-shift evaluation against prototype and simple heuristic baselines.
Boundaries
- Do not add a legacy method-keyed
external_modelscontract if the canonical model catalog/runtime can own the dependency. - Do not send prompts to an external judge without explicit operator configuration and privacy documentation.
- Do not promise latency or accuracy from a single local prototype measurement.
- Do not change existing
rule_name:easy|medium|harddecision references.
Acceptance criteria
- The LLM extractor is optional, typed, and discoverable in diagnostics.
- Invalid or ambiguous output cannot become a confident route.
- Timeout/error/fallback behavior is deterministic and tested.
- Credentials are resolved through the supported provider boundary and are never logged or stored.
- Evaluation reports quality, latency, cost, and multilingual behavior on a representative dataset.
- Existing prototype-only configurations behave unchanged.
Relevant surfaces
src/semantic-router/pkg/classification, src/semantic-router/pkg/config/model_config_types.go, src/semantic-router/pkg/modelruntime, decision evaluation (#2333), docs, and tests.