vllm-project/semantic-router

feat: define shared-resource isolation and diagnostics for multi-tenancy

Open

#2,364 opened on Jul 5, 2026

View on GitHub
 (2 comments) (0 reactions) (1 assignee)Go (699 forks)github user discovery
area/corearea/observabilityenhancementhelp wantedoperationspriority/P1roadmap

Repository metrics

Stars
 (4,293 stars)
PR merge metrics
 (PR metrics pending)

Description

Summary

Define how a shared router safely reuses model, embedding, memory, cache, and telemetry resources across tenants without leaking config or user data.

Parent roadmap: #2287 Parent issue: #2342 Related: #2339, #2345, tenant quota/budget issue Related code: src/semantic-router/pkg/modelruntime, src/semantic-router/pkg/memory, src/semantic-router/pkg/cache, src/semantic-router/pkg/routerreplay, src/semantic-router/pkg/observability

Motivation

The value of shared-router multi-tenancy is avoiding duplicated model loading and infrastructure. That only works if shared resources have explicit isolation boundaries and diagnostics do not expose raw tenant identifiers or cross-tenant state.

Scope

Define isolation and diagnostics for:

  • shared embedding/model runtime assets;
  • semantic cache and memory namespaces;
  • router replay, response store, and vector store tenant scoping;
  • telemetry cardinality and redaction;
  • cache/memory opt-out or per-tenant policy.

Non-goals

  • Do not duplicate every runtime asset per tenant by default.
  • Do not allow cross-tenant cache or memory hits unless explicitly safe.
  • Do not expose raw tenant or user identifiers in public diagnostics.

Acceptance criteria

  • Shared resources have documented tenant scoping rules.
  • Replay/cache/memory diagnostics can prove isolation behavior.
  • Unsafe shared-resource paths fail closed or disable sharing by default.

Contributor guide