vllm-project/semantic-router

feat: add SGLang inference-aware backend adapter

Open

#2,351 opened on Jul 5, 2026

View on GitHub
 (3 comments) (0 reactions) (2 assignees)Go (699 forks)github user discovery
area/model-selectionarea/observabilityenhancementhelp wantedoperationspriority/P2roadmap

Repository metrics

Stars
 (4,293 stars)
PR merge metrics
 (PR metrics pending)

Description

Summary

Add an SGLang-specific adapter for inference-aware backend routing once the engine-neutral backend telemetry contract is defined.

Parent roadmap: #2287 Parent issue: #2332 Depends on: engine-neutral backend telemetry contract Related code: src/semantic-router/pkg/config, src/semantic-router/pkg/selection, src/semantic-router/pkg/latency, deploy/operator, deploy/kubernetes

Motivation

Self-hosted deployments often mix serving engines. SGLang can expose different runtime and scheduling signals from vLLM, so the router needs an adapter boundary that maps SGLang metrics into the same normalized backend policy instead of adding one-off model-selection branches.

Scope

Define and implement SGLang support for:

  • backend/replica identity mapping;
  • queue, active request, latency, health, and memory/cache telemetry ingestion;
  • optional cache-affinity signals if exposed by the deployment;
  • config and operator integration points that do not rename existing vLLM-only fields prematurely;
  • diagnostics for adapter availability and telemetry freshness.

Non-goals

  • Do not require SGLang in default smoke tests.
  • Do not fork the logical model-selection contract for SGLang.
  • Do not add SGLang-specific fields to core selection structures if normalized fields are sufficient.

Acceptance criteria

  • SGLang backends can participate in backend-aware routing through the normalized telemetry contract.
  • Stale/missing SGLang telemetry degrades to existing model-level routing behavior.
  • Docs/examples show how an SGLang deployment opts into the adapter.

Contributor guide