feat: add adaptive thresholding and policy calibration
#2,341 opened on Jul 5, 2026
Repository metrics
- Stars
- (4,293 stars)
- PR merge metrics
- (PR metrics pending)
Description
Summary
Add adaptive thresholding and policy calibration so routing thresholds can be tuned from evaluation, feedback, and confidence evidence instead of remaining fixed static constants.
Parent roadmap: #2287 Related: #2322, #2330, #2333, #2340
Motivation
Static thresholds such as fixed similarity cutoffs can be brittle across domains, tenants, datasets, and recipes. Adaptive calibration can improve route quality, but it needs evaluation guardrails to avoid overfitting or unsafe drift.
Scope
This issue should define and implement calibrated threshold workflows for:
- signal confidence thresholds;
- projection score thresholds;
- decision guard thresholds;
- ambiguity and fallback thresholds;
- per-recipe calibration profiles.
Calibration should be driven by offline eval/replay evidence and should produce explicit candidate config changes or snapshot metadata.
Non-goals
- Do not silently mutate thresholds on the request path.
- Do not introduce adaptive behavior without a decision-level eval guardrail.
- Do not merge this with general Router Learning algorithms; this is policy calibration.
Acceptance criteria
- Threshold calibration inputs and outputs are documented.
- At least one threshold family can be calibrated from an eval dataset.
- Calibrated thresholds are represented as explicit config or snapshot changes.
- Reports show quality, false-positive, false-negative, cost, and regression impacts.