vectorstore: make ingestion cancellable, bounded, and transactional

Summary

Add a root lifecycle, bounded streaming ingestion, status retention, constructor rollback, and transactional reconciliation to vector storage. Owner surfaces: src/semantic-router/pkg/vectorstore/, src/semantic-router/pkg/routerruntime/vectorstore_runtime.go, and src/semantic-router/pkg/config/vectorstore.go.

Current behavior

Active jobs do not receive a cancellable root context, file processing expands into full in-memory text/chunk/vector sets, completed status can grow indefinitely, and multi-store metadata mutations or constructor failures can leave leaked or inconsistent state.

Expected behavior

Jobs and backend work cancel within a deadline, ingestion uses bounded streaming batches and workers, status has a retention policy, and every registry/backend/file mutation either commits coherently or reconciles through an explicit saga state.

Acceptance

• Derive job, embedding, backend, and count-update contexts from a lifecycle root; implement bounded Stop(ctx).
• Stream parse, chunk, embed, and insert through bounded batches with file, worker, queue, and memory limits.
• Apply TTL, cardinality, and persistence policy to completed and failed status.
• Roll back constructor and metadata mutation side effects or persist a reconcilable pending state.
• Make shutdown idempotent and join all cleanup errors.

Validation

• Add permanently blocking embedder/backend, stop-deadline, restart, max-size concurrency, and high-cardinality status tests.
• Fault-inject after every registry, backend, memory, and filesystem side effect, then restart and verify convergence.
• Run cd src/semantic-router && go test -race ./pkg/vectorstore ./pkg/routerruntime.
• Add memory and goroutine bounds to the affected vector ingestion E2E profile.

Related

• Parent audit: #2375
• Runtime generation issue created from RR-01