vllm-project/semantic-router

feat: add vLLM inference-aware backend adapter

Open

#2,350 opened on Jul 5, 2026

View on GitHub
 (1 comment) (0 reactions) (1 assignee)Go (699 forks)github user discovery
area/model-selectionarea/observabilityenhancementhelp wantedoperationspriority/P1roadmap

Repository metrics

Stars
 (4,293 stars)
PR merge metrics
 (PR metrics pending)

Description

Summary

Add the vLLM-specific adapter for inference-aware backend routing, using the engine-neutral backend telemetry contract from #2332 and the backend identity contract issue.

Parent roadmap: #2287 Parent issue: #2332 Depends on: engine-neutral backend telemetry contract Related code: src/semantic-router/pkg/config/model_config_types.go, src/semantic-router/pkg/selection, src/semantic-router/pkg/latency, deploy/operator/controllers/backend_discovery.go, deploy/kubernetes/*/semantic-router-values

Motivation

The router already models self-hosted backends as vllm_endpoints / backend refs, and the operator has VLLMEndpoints discovery. vLLM is the first-class self-hosted engine for the project, so backend-aware routing should support vLLM replica signals without hard-coding those signals into generic selection.

Scope

Implement or design the vLLM adapter for:

  • mapping configured/discovered vLLM endpoints to backend/replica IDs;
  • collecting or ingesting vLLM-compatible metrics for queue depth, active requests, TTFT/TPOT, and health;
  • consuming KV/prefix cache affinity hints when exposed by deployment integration;
  • preserving fallback behavior when vLLM telemetry is missing or stale;
  • surfacing diagnostics through replay/observability.

Non-goals

  • Do not make vLLM telemetry mandatory for external provider routing.
  • Do not block generic backend policy work on vLLM-only fields.
  • Do not replace Kubernetes/service discovery in this issue.

Acceptance criteria

  • vLLM endpoints can be represented as backend-aware candidates under one logical model.
  • vLLM telemetry feeds the normalized backend policy with freshness bounds.
  • Existing vllm_endpoints and operator VLLMEndpoints configs continue to work without telemetry.
  • Tests cover stale/missing telemetry fallback.

Contributor guide