research: compare Router Learning algorithms under a shared benchmark protocol

Summary

Create a maintained benchmark protocol for current and future Router Learning adaptation.strategy implementations before they are considered for production.

Parent roadmap: #2287
Parent learning roadmap: #2238
Related: #2239, #2240, #2308, #2333, #2338, #2393

Audited current-main baseline (2026-07-15)

• routing_sampling is the supported day-zero strategy.
• Protection, per-decision modes, candidate sets, Replay diagnostics, and typed outcomes define the comparison boundary.
• Historical Elo, RL-driven, GMTRouter, lookup-table, session-aware, and personalization names are not supported public Router Learning config. They may be used only as clearly labeled offline baselines when reproducible, not advertised as production strategies.
• Current main does not provide one complete shared strategy leaderboard/protocol.

Scope

1. Define versioned scenario/data interfaces for request context, candidate set, base route, action propensity when available, outcome, delay/censoring, and protection state.
2. Include stationary, drift, delayed/sparse feedback, cold start, low coverage, adversarial/poisoned feedback, session continuity, and multi-replica split-state scenarios.
3. Report:
   • task/route quality and calibration;
   • cost, latency, cache/session switching, and reliability;
   • exploration rate, protection suppressions/rescues, and hard-constraint violations;
   • state size, update cost, hot-path latency/allocation, and recovery;
   • uncertainty, coverage, and unsupported regions.
4. Require static/base selection and routing_sampling baselines; add simple contextual/non-contextual baselines appropriate to each proposal.
5. Support offline Replay/eval projections without requiring raw content by default.
6. Version benchmark data, code, config, model identities, objective definitions, and random seeds.
7. Define graduation gates from offline → observe/shadow → canary; no benchmark alone authorizes production apply mode.

Non-goals

• Do not create a second public learning API.
• Do not compare methods at unmatched candidate sets, constraints, cost, or feedback coverage.
• Do not reduce quality/safety to one opaque reward.
• Do not require every research prototype to enter production.

Acceptance criteria

• One documented protocol runs at least static/base and routing_sampling baselines.
• Metrics, uncertainty, coverage, and failure semantics are machine-readable.
• Protection and hard-constraint violations are first-class failures.
• Dataset leakage, delayed/censored outcomes, poisoning, and drift tests exist.
• New strategy issues state their benchmark integration and graduation criteria.
• Reproduction instructions use maintained repository entrypoints and report provenance.
• Results explicitly state when a strategy is not materially better.

Validation entrypoint

make agent-report ENV=cpu CHANGED_FILES="<space-separated changed files>"