research: predict prompt-plus-output demand for downstream pool admission

Motivation

Investigate whether a pre-dispatch estimate of prompt-plus-output demand improves downstream pool admission, spillover, or sizing. The estimate is a hint for the serving/LB layer after per-query semantic routing, not a capacity solver inside model selection.

Audited upstream baseline (2026-07-15)

• Router Replay and response processing can record actual prompt/completion usage after completion.
• pkg/sessiontelemetry provides bounded historical session aggregates.
• FleetSim workloads/CDFs model token demand offline.
• Current main does not expose a calibrated pre-dispatch total-token predictor with uncertainty and downstream acknowledgement.
• Architecture decision #2513 keeps capacity-aware endpoint selection below semantic routing.

Research scope

1. Define the target and timing precisely: input/prompt tokens, expected output distribution, and total demand before dispatch.
2. Establish simple baselines: prompt length, requested max_tokens cap, route/domain bucket, and session moving average.
3. Train/evaluate only on versioned, leakage-safe replay projections; retain uncertainty and calibration by model/domain/workload.
4. Let semantic routing choose the preferred logical model. If an ordered logical-model fallback contract is later implemented through #2294, version and evaluate it separately.
5. Emit the demand estimate, interval, model/config version, and freshness as a downstream LB/admission hint.
6. Record downstream acknowledgement, actual usage, truncation/cancellation, and residual for calibration.
7. Feed aggregate residuals into offline FleetSim sizing and threshold evaluation.

Boundaries

• Do not choose a pool or solve capacity constraints in pkg/extproc or semantic model selection.
• The estimate cannot override authorization, residency, safety, or context-window limits.
• Requested output caps are upper bounds, not labels for actual output.
• Missing/stale/uncertain predictions fall back to a documented conservative baseline.
• The derived training projection does not require raw prompts/responses by default; existing Router Replay retention remains separately configured.

Acceptance criteria

• Target, censoring, cancellation, and missing-usage semantics are documented.
• Leakage-safe splits and simple baselines are reported with calibration/uncertainty.
• Prediction latency and hot-path allocation are bounded.
• Downstream consumers capability-negotiate the hint and acknowledge use/no-use.
• Replay separates estimate, downstream decision, and actual demand.
• FleetSim evaluation shows whether the estimate materially changes SLO/capacity outcomes.
• E2E tests cover streaming, cancellation, absent usage, partial deployment, and fallback.

Likely change surfaces

src/semantic-router/pkg/sessiontelemetry/, src/semantic-router/pkg/routerreplay/, src/semantic-router/pkg/extproc/, observability/export, src/fleet-sim/fleet_sim/workload/, src/fleet-sim/fleet_sim/routing/, LB integration docs, and tests.

Related: #2294, #2332, #2359, #2513, #2550, #2552.

Validation entrypoint

make agent-report ENV=cpu CHANGED_FILES="<space-separated changed files>"

Follow the reported gates and affected E2E profiles.