feat: add inference-aware backend routing for self-hosted replicas

Summary

Define the engine-neutral telemetry and integration contract for inference-aware endpoint/load balancing below semantic routing in self-hosted GPU deployments.

The semantic router currently selects a preferred logical model per request. A future contract may add ordered logical-model fallbacks, but current main does not expose that as a first-class production contract; upstream-error fallback remains tracked by #2294. Envoy/GIE/the serving LB layer combines the preferred-model contract with fresh replica state to select an eligible endpoint. FleetSim sizes and evaluates the policy offline.

Parent roadmap: #2287

Binding architecture decision

#2513, implemented by merged PR #2514 and recorded in website/docs/proposals/batch-and-capacity-aware-routing.md, establishes that:

• semantic routing remains per-query;
• batch/capacity-aware optimization is not folded into the ext_proc routing decision;
• queue, concurrency, spillover, admission, and endpoint choice are serving/LB concerns below routing;
• FleetSim provides offline sizing and policy evidence.

This issue must not add a replica/capacity solver to pkg/extproc or semantic model selection. Revisiting that ownership requires new evidence and an explicit update to #2513.

Child issues

• #2349 — engine-neutral backend telemetry and replica identity contract
• #2350 — vLLM backend adapter
• #2351 — SGLang backend adapter
• #2352 — ROCm ATOM backend adapter
• #2377 — TensorRT-LLM / Triton TensorRT-LLM adapter
• #2353 — Envoy endpoint-selection integration
• #2359 — FleetSim capacity/policy validation

Related WRP tracks: #2550, #2551, #2552, #2553, #2554, #2555, #2556, #2557, #2559. Logical-model fallback: #2294.

Audited upstream baseline (2026-07-15)

• Current routing selects one logical model and has in-flight, latency, session/affinity, replay, and provider/endpoint configuration surfaces. It does not yet expose a first-class ordered logical-model fallback contract (#2294).
• Backend discovery/configuration does not yet provide one normalized, acknowledged, engine-neutral replica telemetry and LB action contract.
• Telemetry availability differs across vLLM, SGLang, ATOM, and TensorRT-LLM/Triton.
• Router-side observation, modeled estimates, and backend-measured facts must remain distinguishable.

Proposed data flow

request -> per-query semantic router -> preferred logical model + hard constraints/optional hints -> Envoy/GIE/LB -> eligible endpoint/replica -> acknowledgement/outcome

FleetSim consumes trace/telemetry snapshots offline and recommends sizing or LB policy; it does not run in the request hot path.

Engine-neutral observation contract

Versioned, TTL-bound observations may include, when a backend can actually provide them:

• backend/replica/pool identity and serving engine/version;
• deployable model/version and capability;
• health/readiness;
• queue depth and active requests;
• concurrency/utilization/memory pressure;
• KV/prefix-cache affinity or availability indicators;
• TTFT/TPOT/throughput summaries;
• energy/power evidence and its measured/modeled provenance;
• observation timestamp, source, quality/confidence, and expiry.

Adapters must capability-negotiate fields. Missing fields cannot be fabricated or treated as fresh.

Ownership and integration

• Router: preferred logical model, hard safety/authorization/residency/context constraints, and optional SLO/demand hints. Ordered logical-model fallback is a separate future contract coordinated with #2294.
• Telemetry adapters: asynchronous normalized observations; no synchronous engine scrape on the hot path.
• Envoy/GIE/LB: health/capacity/cache-aware endpoint admission and spillover.
• Operator/control plane: pool membership, scaling, rollout, and rollback.
• FleetSim: offline scenario evaluation, sizing, and recommendation evidence.
• Replay/observability: distinguish recommendation, observation, downstream acknowledgement/action, and outcome.

Failure semantics

• Stale/missing telemetry falls back to a documented safe LB mode.
• The LB cannot select a model/backend outside router eligibility or hard constraints.
• Partial deployment must be detectable.
• Capacity or energy preferences cannot override authorization, safety, residency, context compatibility, or health.
• High-cardinality replica data belongs in traces/diagnostics or bounded stores, not unbounded metrics labels.

Acceptance criteria

• Logical model routing and endpoint/LB admission are separate, documented stages.
• Observation and acknowledgement schemas are versioned and engine-neutral.
• Every field carries source, freshness, and support semantics.
• vLLM, SGLang, ATOM, and TensorRT-LLM adapters map only evidence they can obtain.
• No synchronous telemetry fetch or capacity solver is added to ext_proc.
• Envoy/GIE/LB action is testable and reported back when supported.
• Missing/stale/partial telemetry has deterministic fallback behavior.
• Replay distinguishes inferred, modeled, measured, recommended, and acted states.
• Existing external-provider/model-only routing works without backend telemetry.
• FleetSim and E2E tests cover stale data, overload, spillover, partial rollout, and rollback.

Likely change surfaces

src/semantic-router/pkg/config/, src/semantic-router/pkg/inflight/, src/semantic-router/pkg/latency/, src/semantic-router/pkg/extproc/ for metadata/acknowledgement only, src/semantic-router/pkg/routerreplay/, Envoy templates, deploy/operator/, deploy/amd/, src/fleet-sim/, observability, docs, and E2E tests.

Start implementation with make agent-report ENV=cpu CHANGED_FILES="<space-separated changed files>" and follow the reported gates.