vllm-project/semantic-router

feat: add tenant-aware quota, budget, and rate-limit policy

Open

#2,356 opened on Jul 5, 2026

View on GitHub
 (0 comments) (0 reactions) (0 assignees)Go (699 forks)github user discovery
area/corearea/networkingenhancementhelp wantedoperationspriority/P1roadmap

Repository metrics

Stars
 (4,293 stars)
PR merge metrics
 (PR metrics pending)

Description

Summary

Add tenant-aware quota, budget, and rate-limit policy as a production guardrail for shared-router deployments.

Parent roadmap: #2287 Related: #2286, #2342 Related code: src/semantic-router/pkg/config/config.go, src/semantic-router/pkg/ratelimit, src/semantic-router/pkg/authz, src/semantic-router/pkg/extproc, src/semantic-router/pkg/observability

Motivation

The current config has AuthzConfig and RateLimitConfig, and H2 multi-tenancy proposes trusted tenant identity plus isolated config scopes. Production multi-tenant deployments also need cost and usage guardrails: per-tenant request/token limits, budget policy, model access, and diagnostics that do not leak tenant identity.

Scope

Define tenant-aware policy for:

  • trusted tenant/user/group identity inputs from authz;
  • per-tenant request and token quotas;
  • per-model or per-recipe budget limits;
  • cost accounting using model pricing and response usage;
  • fail-open/fail-closed behavior;
  • observability, replay, and dashboard summaries with safe tenant identifiers.

Non-goals

  • Do not rely on client-supplied tenant headers without trusted auth.
  • Do not make quota enforcement part of signal extraction.
  • Do not store raw secrets or raw tenant identifiers in replay.

Acceptance criteria

  • Quota and budget policy composes with the existing authz and ratelimit packages.
  • Per-tenant policy can be evaluated before dispatch and updated through config snapshots.
  • Diagnostics show allowed/throttled/budget-exceeded outcomes safely.
  • Existing single-tenant behavior remains unchanged when the policy is disabled.

Contributor guide