feat: add vLLM inference-aware backend adapter
#2,350 opened on Jul 5, 2026
Repository metrics
- Stars
- (4,293 stars)
- PR merge metrics
- (PR metrics pending)
Description
Summary
Add the vLLM-specific adapter for inference-aware backend routing, using the engine-neutral backend telemetry contract from #2332 and the backend identity contract issue.
Parent roadmap: #2287
Parent issue: #2332
Depends on: engine-neutral backend telemetry contract
Related code: src/semantic-router/pkg/config/model_config_types.go, src/semantic-router/pkg/selection, src/semantic-router/pkg/latency, deploy/operator/controllers/backend_discovery.go, deploy/kubernetes/*/semantic-router-values
Motivation
The router already models self-hosted backends as vllm_endpoints / backend refs, and the operator has VLLMEndpoints discovery. vLLM is the first-class self-hosted engine for the project, so backend-aware routing should support vLLM replica signals without hard-coding those signals into generic selection.
Scope
Implement or design the vLLM adapter for:
- mapping configured/discovered vLLM endpoints to backend/replica IDs;
- collecting or ingesting vLLM-compatible metrics for queue depth, active requests, TTFT/TPOT, and health;
- consuming KV/prefix cache affinity hints when exposed by deployment integration;
- preserving fallback behavior when vLLM telemetry is missing or stale;
- surfacing diagnostics through replay/observability.
Non-goals
- Do not make vLLM telemetry mandatory for external provider routing.
- Do not block generic backend policy work on vLLM-only fields.
- Do not replace Kubernetes/service discovery in this issue.
Acceptance criteria
- vLLM endpoints can be represented as backend-aware candidates under one logical model.
- vLLM telemetry feeds the normalized backend policy with freshness bounds.
- Existing
vllm_endpointsand operatorVLLMEndpointsconfigs continue to work without telemetry. - Tests cover stale/missing telemetry fallback.