vllm-project/semantic-router

router replay: make async writes drainable and retention queries bounded

Open

#2,472 opened on Jul 12, 2026

View on GitHub
 (0 comments) (0 reactions) (0 assignees)Go (699 forks)github user discovery
area/corearea/observabilitybughelp wantedoperationspriority/P1roadmap

Repository metrics

Stars
 (4,293 stars)
PR merge metrics
 (PR metrics pending)

Description

Summary

Give Replay stores one bounded admission, write, close, retention, pagination, and record-size contract. Owner surfaces: src/semantic-router/pkg/routerreplay/recorder.go, src/semantic-router/pkg/routerreplay/store/redis.go, postgres.go, qdrant.go, and milvus.go.

Current behavior

Some asynchronous stores acknowledge work before durable success, errors can be invisible, close does not consistently stop admission and drain, TTL behavior differs by backend, and list operations can scan or materialize retained history. Raw tool fields can also bypass effective record-size limits.

Expected behavior

Every backend follows an explicit open, closing, and closed lifecycle; acknowledged work is durable or observably failed; retention and pagination are backend-conformant; and total serialized record size is bounded.

Acceptance

  • Stop admission atomically, drain the bounded queue, wait for workers, then close clients exactly once.
  • Propagate caller and runtime cancellation; surface asynchronous failures through flush results, metrics, or a documented failure policy.
  • Pass one short-TTL conformance suite in synchronous and asynchronous modes.
  • Use backend-native cursor and limit pagination without full-corpus materialization.
  • Apply total serialized record and tool-trace budgets, including raw fields.

Validation

  • Add barrier tests for enqueue, active work, concurrent add/close, double close, saturation, timeout, and shutdown under -race.
  • Add short-TTL backend conformance and high-cardinality cursor/RSS tests.
  • Run cd src/semantic-router && go test -race ./pkg/routerreplay/....
  • Verify shutdown preserves every acknowledged ID or reports it failed.

Related

  • Parent audit: #2375
  • Replay confidentiality and authorization: #1146
  • Replay list summaries: #2157
  • Restart-safe state history: #1608

Contributor guide