research: learn joint tool-model routing from fleet outcomes

Motivation

The router can already filter/retrieve request-side tools and Router Replay can capture bounded tool traces. The research gap is a normalized outcome contract that can support offline evaluation of tool/model combinations without turning the router into a tool executor.

Audited upstream baseline (2026-07-15)

• pkg/tools provides tool retrieval/filtering primitives and the external-processor path sees the request tool catalog.
• The response path can observe model-emitted tool calls and replay records can include tool trace/outcome fields.
• Current main does not provide a generally trusted label for tool-call success, usefulness, or end-to-end task completion.
• The router does not execute tools; execution results require a gateway/orchestrator feedback contract.
• Router Learning consumes typed experience; Router Memory is not the experience store.

Research scope

1. Define versioned, content-minimized events for:
   • offered/retrieved tools;
   • selected logical model and version/config identity;
   • emitted tool call;
   • execution result supplied by a trusted gateway;
   • optional task/evaluator outcome and provenance.
2. Keep provider failure, invalid tool call, authorization denial, execution failure, and low task utility distinct.
3. Materialize uncertainty-aware tool/model aggregates from Router Replay asynchronously.
4. Establish independent tool-only and model-only baselines before evaluating a joint policy.
5. Evaluate offline first; any online use must remain inside the validated tool/model candidate sets and Router Learning protection.
6. Define delayed/corrected outcome, cold-start, tool-version, and schema-migration behavior.

Boundaries

• Authorization policy always precedes relevance or learned utility.
• A learned policy cannot make a denied tool visible or executable.
• The new derived tool/model learning projection omits raw prompts, tool arguments, and tool results by default. Existing Router Replay may retain bounded raw fields when configured and must be governed separately.
• No synchronous experience-store dependency on the request path.
• This issue does not implement a workflow engine or tool executor.

Acceptance criteria

• Event schemas include tool/model/config version and provenance.
• Trusted execution outcomes cannot be spoofed by ordinary request content.
• Unknown/ambiguous outcomes remain unknown.
• Offline evaluation reports coverage, uncertainty, and simple baselines.
• Derived-projection privacy/cardinality/retention limits and existing Replay redaction/retention interactions are enforced.
• Shadow/canary/rollback and protection behavior are documented before live use.
• Replay can reconstruct why a tool/model recommendation was made.

Likely change surfaces

src/semantic-router/pkg/tools/, src/semantic-router/pkg/extproc/, src/semantic-router/pkg/routerreplay/, src/semantic-router/pkg/routerruntime/, src/semantic-router/pkg/config/, observability, gateway integration docs, and tests.

Related: #2238, #2361, #2544, #2547.

Validation entrypoint

make agent-report ENV=cpu CHANGED_FILES="<space-separated changed files>"

Follow the reported gates and affected E2E profiles.