vLLM Semantic Router 2026-H2 roadmap

vLLM Semantic Router 2026-H2 roadmap

This is the public H2 direction and issue index for 2026-07-01 through 2026-12-31. It is not a promise that every item ships in v0.4.

The v0.4 milestone is a coordination bucket for open, unassigned H2 work until maintainers re-triage it into a narrower release plan.

Upstream audit baseline

The implementation claims and issue coverage in this index were re-audited on 2026-07-15 against origin/main@deb1d7856cd604f7688efebe2e75c3698919dda9 and live GitHub issue metadata. All 139 open issues carrying the roadmap label at that audit point were included in the review; the live count may change after the audit.

Before implementation, contributors must re-run make agent-report ENV=cpu CHANGED_FILES="<space-separated changed files>" (or choose ENV=amd for AMD platform work) because paths, contracts, issue ownership, and upstream capabilities can change after this audit.

Important corrections captured by this audit:

• Router Learning uses global.router.learning.enabled, adaptation.strategy: routing_sampling, protection, per-decision adaptations, Router Replay outcomes, and offline recipe learning. Legacy public lookup_tables and method-keyed bandit/Elo/session APIs are not the current contract.
• Router Memory is prompt-visible user/conversation memory. It is not the Router Learning failure/experience store.
• pkg/authz resolves upstream credentials. It is not the tool RBAC/governance policy engine.
• Per #2513/#2514, semantic routing remains per-query. Capacity, queue, spillover, endpoint, and energy-aware backend decisions belong to Envoy/GIE/LB/serving control planes below it; FleetSim sizes and evaluates them offline.
• FleetSim already contains FleetOptimizer, threshold/Pareto analysis, power models, tokens-per-joule analysis, and grid-flex evaluation. The energy gap is runtime measurement/calibration plus downstream LB policy—not the absence of energy modeling.
• Retention directives already exist in the router. Backend KV-cache enforcement is capability-dependent, and the relevant vLLM work remains an open RFC: https://github.com/vllm-project/vllm/issues/37003
• The router filters request-side tool catalogs and observes response tool traces; it does not execute tools. Guaranteed pre-execution RBAC requires an execution gateway/orchestrator boundary.
• Static Router Flow workflows already support ordered roles and a final model. #1815 now tracks the remaining per-step controls and accounting, not a missing workflow engine.
• Current main has no general cross-encoder runtime or /v1/rerank; PR #2236 is open and unmerged. #2250–#2252 are dependency-ordered follow-ups, not descriptions of shipped behavior.
• Complexity prototype scoring can use local or OpenAI-compatible remote embeddings. #2154 adds an optional generative judge rather than fixing a local-mmBERT-only limitation.
• pkg/authz already supports ext-authz-injected per-user provider keys. #2286 separately scopes caller-credential forwarding to allowlisted trusted gateways.

How to read the live status

• Help wanted / v0.4 coordination is computed here as open + unassigned + v0.4 milestone.
• Owned / carryover means open but assigned, or carried in another milestone.
• Closed issues are not presented as open work.
• GitHub labels can lag assignment; this index uses live assignee/milestone state for the two buckets.
• #2287 is the index. Durable scope changes belong here or in linked issues, not only in comments.

Roadmap principles

• Preserve the per-query pipeline: signals → projections → decisions → strategy/model selection → logical model.
• Keep semantic choice, downstream endpoint admission, and offline fleet planning as separate owned stages.
• Add capability through typed contracts, validation, registries/adapters, provenance, and explicit acknowledgement.
• Separate portable MoM identity, environment binding, resolution lock, and realized traces; credentials, mutable health, and private traces never enter a portable bundle.
• Keep request-path behavior reconstructable through typed diagnostics, configured Router Replay, and content-minimized derived learning/audit projections; do not imply that existing Replay never retains request/response/tool content.
• Guard learning/adaptation with offline evidence, protection, shadow/canary, approval, and rollback.
• Treat authorization, safety, residency, context compatibility, and explicit denials as hard constraints.
• Do not put synchronous global optimization, training, or durable experience reads on the ext_proc hot path.
• Distinguish measured observations, calibrated estimates, model outputs, recommendations, acknowledgements, actions, and outcomes.

Workstreams

1. Config, recipes, control plane, and multi-tenancy

Outcome: Make configuration and MoM artifacts versioned, validated, portable, multi-recipe, multi-tenant, and consistent across API, CLI, dashboard, operator, Helm, and maintained examples.

Help wanted / v0.4 coordination:

• #2286 — feature: safely forward an authenticated caller credential to a trusted upstream gateway
• #2326 — feature: redesign management API around versioned config snapshots
• #2562 — feature: implement the portable, content-addressed MoM Bundle lifecycle
• #2342 — feat: support shared-router multi-tenancy with isolated config scopes
• #2356 — feat: add tenant-aware quota, budget, and rate-limit policy
• #2363 — feat: add tenant-scoped config snapshots and recipe activation
• #2469 — config: enforce version and unknown-field parity across all producers

Owned / carryover:

• #1138 — feature: abstract the API layer for multiple front-end protocols
• #1509 — feature: persist dashboard control-plane and config state in a database
• #1610 — feature: harden deployed config projections and reconcile remaining gaps
• #2331 — feat: support entrypoints and multiple routing recipes
• #2354 — feat: normalize auto model names and algorithm virtual slugs through entrypoints
• #2355 — feat: add operator, CRD, and Helm parity for H2 config contracts
• #2362 — feat: add trusted tenant identity resolution for shared-router multi-tenancy
• #2364 — feat: define shared-resource isolation and diagnostics for multi-tenancy
• #2475 — config: reject non-positive periodic intervals before ticker creation

2. Signals, decisions, provider capability, and model quality

Outcome: Make signal extraction and decision evaluation extensible, calibrated, measurable, and contract-driven across model runtimes.

Help wanted / v0.4 coordination:

• #1485 — bug: default domain classifier underperforms on unseen data
• #2154 — feature: add an optional generative-LLM extractor for complexity signals
• #2247 — feature: Add model-managed cross-encoder reranking for two-stage retrieval/routing
• #2250 — feature: declare cross-encoder rerankers in model_catalog
• #2251 — feature: optional two-stage cross-encoder rerank as a routing signal
• #2252 — feature: support multiple cross-encoder architectures behind a typed runtime
• #2330 — feat: add unified signal registry and observation protocol
• #2335 — feat: expand signal extractors and improve signal model coverage
• #2358 — feat: define provider capability inventory and protocol conformance contract
• #2360 — feat: define signal model lifecycle, datasets, and release provenance
• #2382 — research: define next-generation router base model and model family
• #2393 — feat: define preference-driven recipe objective spec for offline tuning
• #2396 — Modularize native router model bindings and runtime capability contracts

Owned / carryover:

• #1166 — research: evaluate RADAR-inspired difficulty and model-budget routing
• #1422 — research: evaluate PRISM-inspired legitimacy signals before model selection
• #2122 — bug: reject unsupported decision-rule operators during config validation
• #2249 — bug: BERT pooler weight loaded transposed in TraditionalBertClassifier (miscalibrated classification confidence)
• #2333 — feat: add a decision-level routing evaluation harness
• #2341 — feat: add adaptive thresholding and policy calibration
• #2394 — research: evaluate base + multi-LoRA serving for router classifier assets

3. Multimodal and image-routing robustness

Outcome: Make multimodal/image routing correct, calibrated, discoverable, observable, and exercised across Candle/ONNX and HTTP surfaces.

Help wanted / v0.4 coordination:

• #2057 — [Router] Text-bearing image content can over-fire embedding meta-rules without a stage-2 text-modality check
• #2319 — [CI/Build] Model-gated multimodal tests never execute in CI: MULTIMODAL_MODEL_PATH is set nowhere (Go and Rust)

Owned / carryover:

• #2165 — [Router] image-routing.yaml thresholds need recalibration after the encoder-correctness fixes (#1927, #1928, #1943)
• #2166 — [Bindings] onnx-binding image preprocessing diverges from candle-binding's PIL-equivalent resize
• #2317 — [Router] Extend HTTP API image input to classify/batch and the similarity endpoints (deferred from #2104)
• #2318 — [Router] /api/v1/embeddings/models cannot report the multimodal model, so image capability is undiscoverable
• #2347 — feat: harden multimodal and image-routing signal robustness

4. Router Learning, Router Memory, agent budgets, and protection

Outcome: Productionize guarded Router Learning and prompt-visible Router Memory as separate systems, with typed experience, protection, diagnostics, evaluation, approval, and rollback.

Help wanted / v0.4 coordination:

• #2018 — feature: runtime session token-budget enforcement with graduated response ladder
• #2238 — Router Learning adaptation, protection, experience, and recipe-learning roadmap
• #2239 — Harden the Router Learning runtime contract
• #2240 — Build Router Learning experience materialization
• #2241 — Extend Router Learning adaptation composition
• #2242 — cleanup: finish Router Learning migration and reject legacy learning config paths
• #2243 — Define distributed Router Learning state semantics
• #2338 — bench: validate safe exploration and protection policies for agent routing
• #2339 — feat: stabilize prompt-visible Router Memory for production agent routing
• #2340 — feat: add offline agentic config tuning with human and CI approval
• #2346 — research: compare Router Learning algorithms under a shared benchmark protocol
• #2365 — feat: add Router Memory diagnostics and fail-open runtime receipts
• #2366 — feat: define agentic config proposal and patch contract
• #2367 — feat: add approval, activation, and rollback flow for agentic config tuning
• #2541 — research: train and evaluate offline RL for session-length-minimizing routing
• #2542 — feat: add HaluGate-driven per-model per-domain reputation routing
• #2544 — research: route on typed failure experience from Router Replay
• #2548 — research: add fleet-wide caller reputation with cross-agent threat propagation
• #2559 — research: orchestrate closed-loop self-adaptation across WRP knobs

Owned / carryover:

• #1140 — research: learn from typed request and outcome signals without online recipe mutation
• #1512 — feature: build a DSL feedback loop from routing history and user feedback
• #2308 — research: benchmark LinUCB and Linear Thompson as future Router Learning strategies

5. Looper, fallback, multi-model algorithms, and composition

Outcome: Improve multi-model algorithms and fallback/composition without turning the semantic router into a workflow engine.

Help wanted / v0.4 coordination:

• #1815 — feature: add per-step controls and accounting to static Router Flow workflows
• #2337 — research: explore additional Looper-family routing algorithms
• #2471 — looper: lifecycle-own workflow state and make resume semantics real
• #2549 — research: enforce typed cross-request behavioral commitments

Owned / carryover:

• #1141 — research: forecast serving demand for downstream admission and scaling
• #2294 — feature: upstream-error model fallback (failover) through the semantic router
• #2336 — feat: optimize and harden existing Looper, Flow, Fusion, and ReMoM algorithms
• #2343 — feat: design hierarchical routing and recipe chaining without becoming a workflow engine

6. Self-hosted inference, downstream LB, FleetSim, and performance

Outcome: Join per-query logical-model routing to engine-neutral observations and downstream LB/control-plane decisions, with FleetSim providing offline capacity, SLO, cost, and energy evidence.

Help wanted / v0.4 coordination:

• #2332 — feat: add inference-aware backend routing for self-hosted replicas
• #2372 — perf: analyze and optimize ROCm production performance for high concurrency and long context
• #2550 — feat: add follow-the-sun pool-boundary and capacity recommendations
• #2551 — research: predict prompt-plus-output demand for downstream pool admission
• #2552 — research: evaluate capacity-aware cascade admission below semantic routing
• #2554 — research: forecast workload archetypes for proactive per-pool scaling
• #2555 — research: co-optimize FleetSim compression band and pool boundary
• #2556 — research: provision fleets for mixed workload archetype distributions
• #2557 — research: evaluate energy-aware backend/LB policy from FleetSim power models

Owned / carryover:

• #1510 — feature: establish large-scale performance regression coverage across Candle/ONNX, NVIDIA, and AMD
• #2349 — feat: define engine-neutral backend telemetry and replica identity contract
• #2350 — feat: add vLLM inference-aware backend adapter
• #2351 — feat: add SGLang inference-aware backend adapter
• #2352 — feat: add ROCm ATOM inference-aware backend adapter
• #2353 — feat: add Envoy endpoint-selection integration for backend-aware routing
• #2359 — bench: connect fleet-sim capacity modeling to routing policy validation
• #2376 — ops: revisit Docker and Kubernetes backend endpoint support completeness
• #2377 — feat: add TensorRT-LLM and Triton TensorRT-LLM inference-aware backend adapter
• #2395 — perf: define MIGraphX-first ONNX Runtime provider strategy for AMD router signal models
• #2553 — feat: bridge existing retention directives to backend KV-cache adapters

7. Observability, dashboard, replay, streaming, and adoption

Outcome: Make decisions inspectable and adoptable through bounded replay, diagnostics, dashboards, headers, recipes, tutorials, and onboarding.

Help wanted / v0.4 coordination:

• #1153 — feature: Model onboarding procedure
• #2369 — feat: add frontend routing analytics views for dashboard
• #2370 — feat: add drift and unrouted-query clustering with privacy controls
• #2378 — recipes: tune out-of-box presets for balance, cost, accuracy, speed, privacy, and grounding
• #2472 — router replay: make async writes drainable and retention queries bounded

Owned / carryover:

• #2200 — feature: redesign VSR response header contract for v0.4
• #2334 — docs: add reference recipes and tutorials for common routing patterns
• #2344 — research: explore streaming semantic routing on partial input
• #2345 — feat: add dashboard analytics for routing quality, drift, and unrouted traffic
• #2368 — feat: add backend analytics read model for routing quality dashboards

8. Security, code quality, lifecycle, and API hardening

Outcome: Close known security and resource-lifecycle gaps, make trust boundaries explicit, and keep diagnostics content-free by default.

Help wanted / v0.4 coordination:

• #2357 — security: track H2 router security hardening across auth, feedback, and Looper paths
• #2463 — security: authenticate and isolate the router management API
• #2464 — security: make router diagnostics content-free by default
• #2465 — security: remove reusable browser session tokens from dashboard URLs
• #2467 — security: replace ML pipeline filesystem paths with owned job and artifact handles
• #2470 — runtime: introduce generation-owned construction, reload, and graceful shutdown
• #2474 — vectorstore: make ingestion cancellable, bounded, and transactional
• #2491 — fix: make the non-CGO Candle backend fail closed or contract-faithful
• #2543 — research: add cumulative multi-turn and per-caller risk scoring
• #2547 — feat: enforce request-level tool RBAC across router and execution gateway
• #2558 — feat: extend the routing DSL with WRP governance-as-code constraints

Owned / carryover:

• #1146 — security: restrict Router Replay read APIs with explicit authorization
• #1388 — security: block SSRF in dashboard OpenWeb fetches
• #1443 — security: looper header injection allows bypassing all security plugins
• #1445 — security: identity headers can be spoofed without ext_authz
• #1452 — security: authenticate and rate-limit Router Learning outcome ingestion
• #1456 — security: cap Looper fan-out and total upstream-call amplification
• #2222 — bug: possible memory leak or unbounded retention under load in vLLM Semantic Router v0.3
• #2375 — quality: audit and harden router code quality, security, and API boundaries
• #2466 — security: make dashboard API authorization route-bound and deny by default
• #2473 — cache: propagate cancellation and return similarity per lookup
• #2476 — security: add reproducible dependency and toolchain vulnerability gates
• #2478 — api: bound and type external RAG request templates and responses
• #2482 — dashboard: make credential lifecycle audit events transactionally durable

9. Tools, MCP, agents, and environment integrations

Outcome: Define tool/MCP/agent contracts while keeping authorization, relevance, execution, memory, and workflow orchestration at explicit seams.

Help wanted / v0.4 coordination:

• #2361 — feat: explore tool- and MCP-aware routing contracts
• #2468 — security: constrain OpenClaw image and skill provisioning inputs
• #2545 — research: learn joint tool-model routing from fleet outcomes
• #2546 — research: coordinate agent memory tiers, tool surfaces, and scheduler hints at the gateway

Owned / carryover:

• #1521 — feature: support full Matrix-style WebSocket collaboration between ClawOS rooms
• #1522 — research: study context management patterns and OpenClaw best practices for ClawOS
• #1524 — research: explore sociology-inspired environment-aware ClawOS modes beyond team mode

10. Edge deployment

Outcome: Support edge AI systems with explicit local/external model capability, privacy, fallback, and operational constraints.

Help wanted / v0.4 coordination:

• None at this audit point.

Owned / carryover:

• #2373 — feat: support AMD AI PC edge deployments with local and external model routing
• #2374 — feat: support NVIDIA DGX Spark edge deployments with local and external model routing

11. Testing, release, and validation

Outcome: Keep E2E, Docker/Kubernetes, native-binding compilation, and release validation coherent as H2 features expand.

Help wanted / v0.4 coordination:

• #2379 — test: revisit E2E framework and testcase matrix across Docker and Kubernetes
• #2477 — bindings/CI: restore ONNX Go test compilation and make it mandatory

Owned / carryover:

• #1519 — feature: expand module-level end-to-end coverage, especially dashboard workflows

12. Roadmap coordination

Outcome: Keep issue coverage, live ownership, milestones, split/superseded relationships, and upstream implementation assumptions auditable.

Help wanted / v0.4 coordination:

• #2287 — vLLM Semantic Router 2026-H2 roadmap

Owned / carryover:

• None at this audit point.

Workload–Router–Pool paper follow-ups

Paper: https://arxiv.org/abs/2603.21354

The paper's opportunities are tracked below, but two different classifications must remain separate:

• Paper tier is the paper's own conclusion: opportunities 2, 5, 8, 11, 15, and 20 are engineering; the other 15 are research.
• Current repository baseline describes what exists on audited main and therefore what the first implementation gate should be.

An engineering paper tier does not mean production actuation is already implemented or approved.

#	Opportunity	Paper tier	Current repository baseline	Canonical tracker	First repository gate
1	Offline RL for session objectives	Research	Research	#2541	Versioned Replay-derived trajectory + supervised/bandit baselines
2	HaluGate model/domain reputation	Engineering	Data-contract follow-up	#2542	Model/config identity + async materializer + shadow
3	Cumulative session/caller risk	Research	Security research	#2543	Trusted identity + bounded local accumulator
4	Failure-class-aware routing	Research	Research	#2544	Typed outcomes from Router Replay; not Router Memory
5	Runtime session token budget	Engineering	Engineering follow-up	#2018	Pre-dispatch reservation + post-response reconciliation
6	Joint tool/model learning	Research	Research	#2545	Trusted execution-outcome contract
7	Gateway-coordinated agent context	Research	Integration research	#2546	Versioned ownership/precedence envelope
8	Request-level tool RBAC	Engineering	Security engineering	#2547	Request filtering + execution-gateway re-authorization
9	Shared caller risk evidence	Research	Security research	#2548	Explicit trust domain, privacy, attestation, poisoning study
10	Cross-request behavioral commitments	Research	Security research	#2549	Closed typed state machine; no free-form executable policy
11	Follow-the-sun rebalancing	Engineering	Engineering concept	#2550	Observe/recommend first; downstream actuation later
12	Pool-state inference/backend awareness	Research	Engineering contract work	#2332 and #2349	Engine-neutral telemetry + LB acknowledgement
13	Prompt-plus-output demand hint	Research	Research	#2551	Calibrated uncertainty + downstream admission experiment
14	Capacity-aware cascade admission	Research	Research	#2552	FleetSim matched-cost/SLO evaluation below routing; fallback depends on #2294
15	KV-cache retention bridge	Engineering	Partial router baseline	#2553	Capability negotiation + explicit unsupported/no-op behavior
16	Workload-archetype forecast	Research	Research	#2554	Privacy-safe schema + simple forecast baselines
17	Compression/pool co-adaptation	Research	Research with FleetSim baseline	#2555	Calibrate runtime compression knobs to FleetSim C&R gamma
18	Mixed-archetype provisioning	Research	Research with FleetSim baseline	#2556	Mixture schema in existing core/workload/routing/optimizer tree
19	Energy-aware backend policy	Research	Research with FleetSim power baseline	#2557	Runtime energy provenance/calibration + downstream LB policy evaluation
20	WRP governance as code	Engineering	Engineering/design	#2558	Typed ownership-aware compiler; hard constraints first
21	Closed-loop WRP coordination	Research	Capstone research	#2559	Separate router and LB/operator actors with independent gates

Recommended dependency order

1. Trust and contracts: #2362, #2349, #2547, #2553, #2558.
2. Typed observations/materializers: #2542, #2543, #2544, #2545, #2551, #2557.
3. Offline baselines and simulation: #2541, #2359, #2552, #2554, #2555, #2556.
4. Observe/recommend integrations: #2018, #2332, #2546, #2549, #2550.
5. Capstone only after stable actors: #2548 and #2559.

Cross-cutting done criteria

• Every adaptive opportunity has a reproducible fixed/simple baseline and uncertainty/coverage report.
• Replay distinguishes source, freshness, confidence, recommendation, acknowledgement/action, fallback, and outcome.
• Observe/shadow and safe degradation precede canary or automatic operation.
• Identity, privacy, isolation, retention, poisoning, and secret-redaction boundaries are explicit.
• Pool work validates stale telemetry, cold start, transition safety, hard constraints, SLO, rollback, and partial deployment.
• Config/policy changes compose with snapshots, validation, evaluation, approval, canary, and rollback.
• External capability dependencies are versioned and capability-negotiated; an open upstream RFC is not treated as a stable API.

Superseded inputs

• #2322 was split into focused issues and closed.
• Earlier community comments were split into decision-level evaluation, agentic config tuning, fallback, reference recipe/tutorial, and multi-tenancy tracks.
• Closed child issues remain discoverable through their parents but are not kept in open-work buckets.

Acceptance criteria for this roadmap

• Every open GitHub issue carrying the roadmap label is listed in one workstream.
• Every open, unassigned roadmap issue in the v0.4 coordination bucket carries help wanted; deliberately re-triaged older milestones remain carryover.
• Parent issues link children, current architecture boundaries, likely code surfaces, and validation entrypoints.
• Exploratory work states baseline, hypothesis, dependencies, non-goals, metrics, and graduation gate.
• Production-facing work includes observability, fallback, security/safety, partial-deployment behavior, and rollback.
• Implementation claims are periodically re-audited against current upstream main and external dependency status.