research: train and evaluate offline RL for session-length-minimizing routing

Motivation

The paper motivates optimizing end-to-end session outcomes rather than only per-turn routing quality. The repository can investigate this as an offline Router Learning research track, but current main does not provide a production offline-RL policy or a lookup-table experience API.

Audited upstream baseline (2026-07-15)

• Router Replay records routing context, selected model, usage/cost, HaluGate fields, tool traces, and typed outcome updates.
• pkg/sessiontelemetry maintains bounded in-process session aggregates.
• Router Learning currently exposes routing_sampling plus a protection envelope for bounded online model selection.
• Offline recipe learning is the intended seam for policy changes beyond those online adaptations.
• Legacy lookup_tables are rejected; Router Memory is unrelated prompt-visible conversation memory.

Research scope

1. Define a versioned, content-minimized trajectory materializer from Router Replay and typed outcomes.
2. Define the session objective precisely: success/safety first, then total tokens/cost/turns/latency as constrained objectives.
3. Establish supervised/contextual-bandit and static-router baselines before offline RL.
4. Add counterfactual/off-policy evaluation (OPE) only where action propensities and coverage are sufficient; report unsupported regions instead of extrapolating silently.
5. Train offline and emit a reviewable recipe/policy candidate. Do not update the live router directly.
6. Replay the candidate in FleetSim or a trace harness, then run shadow/canary evaluation under Router Learning protection.

Non-goals

• No synchronous training or durable-store read on the request path.
• No public lookup-table or method-keyed bandit/Elo API.
• No optimization that trades away authorization or safety constraints.
• No claim that historical traces identify the causal effect of unobserved actions.

Acceptance criteria

• Dataset schema includes policy/config version, candidate set, action, propensity when available, outcome provenance, and redaction rules.
• Leakage and train/test split rules are documented at caller/session/time boundaries.
• Baselines and uncertainty intervals accompany every result.
• Unsupported OPE coverage fails closed.
• Candidate policy is reproducible, reviewable, reversible, and compatible with protection.
• The derived trajectory dataset does not require raw prompt/response/tool content by default; existing Router Replay retention remains separately configured.
• Offline, shadow, and canary gates are documented.

Likely change surfaces

src/semantic-router/pkg/routerreplay/, src/semantic-router/pkg/sessiontelemetry/, src/semantic-router/pkg/routerruntime/, src/vllm-sr/cli/, src/fleet-sim/, benchmark assets, docs, and tests.

Related: #2238, #2241, #2243, #2393.

Validation entrypoint

make agent-report ENV=cpu CHANGED_FILES="<space-separated changed files>"

Follow the reported gates and affected E2E profiles.