vllm-project/semantic-router

feat: add Envoy endpoint-selection integration for backend-aware routing

Open

#2,353 opened on Jul 5, 2026

View on GitHub
 (0 comments) (0 reactions) (1 assignee)Go (699 forks)github user discovery
area/model-selectionarea/networkingenhancementhelp wantedoperationspriority/P1roadmap

Repository metrics

Stars
 (4,293 stars)
PR merge metrics
 (PR metrics pending)

Description

Summary

Define how backend-aware routing decisions are enforced at the Envoy/upstream endpoint boundary after the router selects a logical model and backend/replica candidate.

Parent roadmap: #2287 Parent issue: #2332 Related code: src/semantic-router/pkg/extproc, src/vllm-sr/cli/templates/envoy.template.yaml, deploy/operator/controllers/semanticrouter_envoy.go, deploy/operator/controllers/backend_discovery.go

Motivation

Current Envoy integration lets the router mutate request metadata and logical model routing, while Envoy generally performs endpoint load balancing inside the selected upstream cluster. Backend-aware routing needs an explicit control boundary; otherwise the router cannot guarantee queue-depth, GPU-load, or KV-cache-affinity placement.

Scope

Evaluate and implement the least invasive endpoint-control mechanism:

  • router-selected backend ref / endpoint metadata emitted to Envoy;
  • Envoy cluster or subset routing configuration generated by CLI/operator templates;
  • optional feedback path where Envoy reports the chosen upstream endpoint back to the router;
  • behavior for unsupported gateway profiles;
  • replay/trace fields for requested backend, actual backend, and fallback reason.

Non-goals

  • Do not require all gateways to support deterministic endpoint pinning on day one.
  • Do not make endpoint pinning bypass health checks or load-shedding.
  • Do not hide endpoint-selection fallback from diagnostics.

Acceptance criteria

  • The router/Envoy contract for backend-aware routing is explicit and testable.
  • Envoy config generation can support backend refs or subset routing without breaking existing model-level routing.
  • Replay can show requested backend, actual backend when known, and fallback reason.

Contributor guide