vllm-project/semantic-router

cache: propagate cancellation and return similarity per lookup

Open

#2,473 opened on Jul 12, 2026

View on GitHub
 (0 comments) (0 reactions) (1 assignee)Go (699 forks)github user discovery
area/corearea/observabilitybughelp wantedpriority/P1roadmap

Repository metrics

Stars
 (4,293 stars)
PR merge metrics
 (PR metrics pending)

Description

Summary

Make semantic-cache operations context-aware and return similarity as part of each lookup result. Owner surfaces: src/semantic-router/pkg/cache/, src/semantic-router/pkg/extproc/req_filter_cache.go, and src/semantic-router/pkg/extproc/recorder.go.

Current behavior

Cache interfaces discard request and runtime contexts, several backends use background contexts, and similarity is published through cache-global mutable state. A concurrent lookup or a miss can therefore expose another request's score in headers, telemetry, or Replay.

Expected behavior

Cancellation reaches storage and embedding work, each lookup returns its own body/found/similarity value, misses and errors have explicit absent similarity, and partial construction cleans up clients.

Acceptance

  • Pass context through health, read, write, and search interfaces and all backends.
  • Return a value object containing body, found, and similarity; remove global last-similarity state.
  • Return zero or absent similarity on misses and errors.
  • Preserve request-owned similarity through debug and Replay paths.
  • Close every partially constructed client on failure.

Validation

  • Add barrier-controlled concurrent lookups proving result isolation.
  • Add miss, error, cancellation, native/backend interruption, and constructor-failure tests.
  • Run cd src/semantic-router && go test -race ./pkg/cache/... ./pkg/extproc.
  • Include a cancellation and per-request-score assertion in the affected cache E2E profile.

Related

  • Parent audit: #2375
  • Memory growth tracker: #2222

Contributor guide