vllm-project/semantic-router

feat: add follow-the-sun pool-boundary and capacity recommendations

Open

#2,550 opened on Jul 15, 2026

View on GitHub
 (0 comments) (0 reactions) (0 assignees)Go (699 forks)github user discovery
area/bencharea/model-selectionarea/observabilityenhancementevaluationhelp wantedoperationspriority/P1roadmap

Repository metrics

Stars
 (4,293 stars)
PR merge metrics
 (PR metrics pending)

Description

Motivation

Use recent, content-free workload demand to recommend when capacity or a pool boundary should move across regions/time zones. The safe first deliverable is an observable recommendation loop; production actuation belongs to the downstream load-balancing/serving control plane.

Audited upstream baseline (2026-07-15)

  • FleetSim already provides FleetOptimizer, threshold/Pareto optimization, trace/CDF inputs, and fleet/pool models.
  • Those inputs are primarily offline/static; current main does not operate a production sliding-window regional controller.
  • Architecture decision #2513 (implemented by merged PR #2514) keeps semantic routing per-query and puts batch/capacity-aware endpoint selection below it, sized offline by FleetSim.
  • #2332 tracks inference-aware backend routing integration.

Phased scope

Phase 1 — observe and recommend

  1. Define a privacy-safe regional demand window using token/request/latency aggregates; no prompt content.
  2. Specify event-time, late data, clock skew, restart, and multi-replica aggregation semantics.
  3. Convert a versioned window snapshot to FleetSim-compatible demand inputs.
  4. Produce a recommendation with target pool/region capacity, confidence, horizon, objective values, and reason.
  5. Compare against static capacity and simple time-of-day baselines in replay/simulation.
  6. Expose dashboards/alerts and operator acknowledgement; no automatic mutation.

Phase 2 — downstream actuation contract

  1. Define an idempotent, authenticated recommendation/acknowledgement API for Envoy/GIE/LB/operator consumers.
  2. Add cooldown, hysteresis, minimum dwell time, rate limits, capacity/SLO/safety constraints, rollback, and manual override.
  3. Roll out in shadow, advise-only, canary, then optional automatic mode.
  4. Record recommendation, downstream decision, actuation, and measured outcome separately.

Architectural boundary

  • Do not add a capacity solver to pkg/extproc or semantic model selection.
  • The semantic router may emit logical model/SLO hints; the downstream LB/control plane chooses an eligible endpoint/pool.
  • Changing that responsibility requires revisiting #2513 explicitly.
  • Cross-region data residency, authorization, and safety constraints are hard constraints, not objective weights.

Acceptance criteria

  • Demand window and FleetSim input contracts are versioned and reproducible.
  • Recommendation quality is measured against simple baselines with uncertainty.
  • Missing/stale telemetry yields no unsafe actuation.
  • Actuation is downstream, idempotent, rate-limited, reversible, and auditable.
  • Residency/SLO/capacity constraints fail closed.
  • Replay distinguishes recommendation from actual downstream action.
  • E2E tests cover partial deployment, stale data, oscillation, rollback, and manual override.

Likely change surfaces

src/fleet-sim/fleet_sim/workload/, src/fleet-sim/fleet_sim/optimizer/, src/fleet-sim/fleet_sim/core/, router observability/export, deployment LB/operator integration, dashboards, docs, and tests.

Related: #2332, #2359, #2551, #2554, #2559.

Contributor guide