cache: propagate cancellation and return similarity per lookup
#2,473 opened on Jul 12, 2026
Repository metrics
- Stars
- (4,293 stars)
- PR merge metrics
- (PR metrics pending)
Description
Summary
Make semantic-cache operations context-aware and return similarity as part of each lookup result. Owner surfaces: src/semantic-router/pkg/cache/, src/semantic-router/pkg/extproc/req_filter_cache.go, and src/semantic-router/pkg/extproc/recorder.go.
Current behavior
Cache interfaces discard request and runtime contexts, several backends use background contexts, and similarity is published through cache-global mutable state. A concurrent lookup or a miss can therefore expose another request's score in headers, telemetry, or Replay.
Expected behavior
Cancellation reaches storage and embedding work, each lookup returns its own body/found/similarity value, misses and errors have explicit absent similarity, and partial construction cleans up clients.
Acceptance
- Pass context through health, read, write, and search interfaces and all backends.
- Return a value object containing body, found, and similarity; remove global last-similarity state.
- Return zero or absent similarity on misses and errors.
- Preserve request-owned similarity through debug and Replay paths.
- Close every partially constructed client on failure.
Validation
- Add barrier-controlled concurrent lookups proving result isolation.
- Add miss, error, cancellation, native/backend interruption, and constructor-failure tests.
- Run
cd src/semantic-router && go test -race ./pkg/cache/... ./pkg/extproc. - Include a cancellation and per-request-score assertion in the affected cache E2E profile.
Related
- Parent audit: #2375
- Memory growth tracker: #2222