research: evaluate RADAR-inspired difficulty and model-budget routing

Summary

Evaluate whether a RADAR-inspired difficulty/ability model improves routing among approved model-and-reasoning-budget configurations.

Research source

RADAR: Reasoning–Ability and Difficulty-Aware Routing for Reasoning LLMs was published as a MATH-AI 2025 workshop poster. The paper models query difficulty and model-budget ability with an item-response-theory-inspired formulation and reports experiments on eight reasoning benchmarks, including out-of-distribution evaluation.

Those are paper results, not evidence that the method improves this repository's workloads. The prior issue text's “30–50% cost reduction,” exact onboarding sample count, and benchmark list were not supported as repository acceptance claims and are removed.

Audited current-main baseline (2026-07-15)

• The router already has typed signals, decisions, candidate model selection, Router Replay/eval surfaces, and bounded Router Learning.
• Current main has no RADARSelector, no production query-difficulty model, and no model-budget ability registry.
• Online Router Learning may change only the selected model inside the configured candidate set and protection envelope. Recipe and signal changes remain offline.

Research scope

1. Reproduce the paper formulation on a versioned benchmark before designing runtime APIs.
2. Define model configuration identity precisely: logical model, deployable version, reasoning controls, cost, and evaluation provenance.
3. Train/evaluate query-difficulty and model-budget ability estimates with leakage-safe in-distribution and out-of-distribution splits.
4. Compare static rules, existing selectors, routing_sampling, and simple calibrated classifiers at matched quality/cost.
5. Test calibration, abstention, unsupported-domain behavior, new-model onboarding, drift, and adversarial inputs.
6. If evidence warrants integration, expose difficulty as a typed signal or a future adaptation.strategy implementation—without adding a legacy standalone selector API.
7. Keep learning/training offline; a production candidate must pass the Router Learning benchmark/protection contract.

Non-goals

• Do not assume model self-estimated difficulty is trusted ground truth.
• Do not update IRT parameters synchronously from unverified production feedback.
• Do not hard-code “easy/medium/hard” model examples or universal cost savings.
• Do not bypass authorization, safety, residency, context, or configured candidate constraints.

Acceptance criteria

• Paper reproduction and repository-specific benchmark are separate and reproducible.
• Dataset/model/config versions and label provenance are recorded.
• OOD, calibration, abstention, latency, cost, and quality are reported with simple baselines.
• New-model onboarding reports uncertainty and required evaluation coverage.
• Any runtime integration uses current signal/Router Learning contracts and observe mode first.
• Results state when RADAR does not outperform simpler approaches.

Validation entrypoint

make agent-report ENV=cpu CHANGED_FILES="<space-separated changed files>"