vllm-project/semantic-router
View on GitHubfeat: add follow-the-sun pool-boundary and capacity recommendations
Open
#2,550 opened on Jul 15, 2026
area/bencharea/model-selectionarea/observabilityenhancementevaluationhelp wantedoperationspriority/P1roadmap
Repository metrics
- Stars
- (4,293 stars)
- PR merge metrics
- (PR metrics pending)
Description
Motivation
Use recent, content-free workload demand to recommend when capacity or a pool boundary should move across regions/time zones. The safe first deliverable is an observable recommendation loop; production actuation belongs to the downstream load-balancing/serving control plane.
Audited upstream baseline (2026-07-15)
- FleetSim already provides
FleetOptimizer, threshold/Pareto optimization, trace/CDF inputs, and fleet/pool models. - Those inputs are primarily offline/static; current
maindoes not operate a production sliding-window regional controller. - Architecture decision #2513 (implemented by merged PR #2514) keeps semantic routing per-query and puts batch/capacity-aware endpoint selection below it, sized offline by FleetSim.
- #2332 tracks inference-aware backend routing integration.
Phased scope
Phase 1 — observe and recommend
- Define a privacy-safe regional demand window using token/request/latency aggregates; no prompt content.
- Specify event-time, late data, clock skew, restart, and multi-replica aggregation semantics.
- Convert a versioned window snapshot to FleetSim-compatible demand inputs.
- Produce a recommendation with target pool/region capacity, confidence, horizon, objective values, and reason.
- Compare against static capacity and simple time-of-day baselines in replay/simulation.
- Expose dashboards/alerts and operator acknowledgement; no automatic mutation.
Phase 2 — downstream actuation contract
- Define an idempotent, authenticated recommendation/acknowledgement API for Envoy/GIE/LB/operator consumers.
- Add cooldown, hysteresis, minimum dwell time, rate limits, capacity/SLO/safety constraints, rollback, and manual override.
- Roll out in shadow, advise-only, canary, then optional automatic mode.
- Record recommendation, downstream decision, actuation, and measured outcome separately.
Architectural boundary
- Do not add a capacity solver to
pkg/extprocor semantic model selection. - The semantic router may emit logical model/SLO hints; the downstream LB/control plane chooses an eligible endpoint/pool.
- Changing that responsibility requires revisiting #2513 explicitly.
- Cross-region data residency, authorization, and safety constraints are hard constraints, not objective weights.
Acceptance criteria
- Demand window and FleetSim input contracts are versioned and reproducible.
- Recommendation quality is measured against simple baselines with uncertainty.
- Missing/stale telemetry yields no unsafe actuation.
- Actuation is downstream, idempotent, rate-limited, reversible, and auditable.
- Residency/SLO/capacity constraints fail closed.
- Replay distinguishes recommendation from actual downstream action.
- E2E tests cover partial deployment, stale data, oscillation, rollback, and manual override.
Likely change surfaces
src/fleet-sim/fleet_sim/workload/, src/fleet-sim/fleet_sim/optimizer/, src/fleet-sim/fleet_sim/core/, router observability/export, deployment LB/operator integration, dashboards, docs, and tests.
Related: #2332, #2359, #2551, #2554, #2559.