feat: add Envoy endpoint-selection integration for backend-aware routing
#2,353 opened on Jul 5, 2026
Repository metrics
- Stars
- (4,293 stars)
- PR merge metrics
- (PR metrics pending)
Description
Summary
Define how backend-aware routing decisions are enforced at the Envoy/upstream endpoint boundary after the router selects a logical model and backend/replica candidate.
Parent roadmap: #2287
Parent issue: #2332
Related code: src/semantic-router/pkg/extproc, src/vllm-sr/cli/templates/envoy.template.yaml, deploy/operator/controllers/semanticrouter_envoy.go, deploy/operator/controllers/backend_discovery.go
Motivation
Current Envoy integration lets the router mutate request metadata and logical model routing, while Envoy generally performs endpoint load balancing inside the selected upstream cluster. Backend-aware routing needs an explicit control boundary; otherwise the router cannot guarantee queue-depth, GPU-load, or KV-cache-affinity placement.
Scope
Evaluate and implement the least invasive endpoint-control mechanism:
- router-selected backend ref / endpoint metadata emitted to Envoy;
- Envoy cluster or subset routing configuration generated by CLI/operator templates;
- optional feedback path where Envoy reports the chosen upstream endpoint back to the router;
- behavior for unsupported gateway profiles;
- replay/trace fields for requested backend, actual backend, and fallback reason.
Non-goals
- Do not require all gateways to support deterministic endpoint pinning on day one.
- Do not make endpoint pinning bypass health checks or load-shedding.
- Do not hide endpoint-selection fallback from diagnostics.
Acceptance criteria
- The router/Envoy contract for backend-aware routing is explicit and testable.
- Envoy config generation can support backend refs or subset routing without breaking existing model-level routing.
- Replay can show requested backend, actual backend when known, and fallback reason.