perf: define MIGraphX-first ONNX Runtime provider strategy for AMD router signal models

Summary

Define and validate a MIGraphX-first ONNX Runtime execution-provider strategy for vLLM Semantic Router's AMD router signal models.

Parent roadmap: #2287 Related: #2372, #2332, #2352, #2359, #1510, #2382 Related code: onnx-binding, onnx-binding/ort-ck-flash-attn, src/vllm-sr/Dockerfile.rocm, tools/docker/Dockerfile.extproc-rocm, tools/make/build-run-test.mk, tools/make/models.mk, e2e/config/onnx-binding, src/semantic-router/pkg/classification, src/semantic-router/pkg/embedding, src/semantic-router/pkg/extproc, src/fleet-sim

Motivation

The AMD production image currently builds onnx-binding with rocm-dynamic, installs onnxruntime_rocm-1.22.1, and uses ROCm EP plus the CK FlashAttention custom-op path for selected ONNX artifacts. The code also contains partial MIGraphX support, but the provider order is inconsistent across classifier and embedding paths:

• classifier path tries MIGraphX first under the ROCm feature, then ROCm EP fallback;
• embedding path tries ROCm EP first and only tries MIGraphX when the migraphx feature is enabled;
• the ROCm Dockerfile builds rocm-dynamic, not migraphx-dynamic.

This needs a deliberate H2 decision because ONNX Runtime's ROCm EP has been removed starting in ORT 1.23, and ORT/AMD recommend migrating AMD ONNX inference workloads to the MIGraphX Execution Provider. Since vLLM SR primarily runs these router-owned signal models inline on AMD ROCm servers, the default provider policy should be evidence-based, explicit, and benchmarked on the actual signal workloads.

Scope

Design and validate:

• the default AMD provider order for onnx-binding (MIGraphX -> ROCm legacy fallback -> CPU, or another measured policy);
• whether the ROCm image should move from rocm-dynamic to migraphx-dynamic;
• consistent provider-selection behavior across classifier, token-classifier, embedding, multimodal, and future reranker paths;
• how CK FlashAttention custom-op artifacts interact with MIGraphX and ROCm EP;
• artifact naming and metadata for portable ONNX, fp16 ONNX, and ROCm/MIGraphX optimized variants;
• cold-start compile/cache behavior, warm latency, tail latency, throughput, VRAM, and correctness/logit drift;
• operator coverage gaps and fallback diagnostics when MIGraphX cannot own the whole graph.

Non-goals

• Do not assume MIGraphX is always faster without benchmark evidence.
• Do not remove ROCm EP fallback until compatibility and performance data justify it.
• Do not turn router signal inference into an external model-serving dependency.
• Do not replace the broader ROCm production-performance analysis in #2372.

Benchmark matrix

At minimum, compare:

• mmbert32k intent classifier;
• PII token classifier;
• factcheck and feedback classifiers;
• mmBERT embedding / 2D Matryoshka variants;
• long-context and high-concurrency request mixes;
• CK FlashAttention optimized artifacts where available.

Provider/runtime variants:

• ORT ROCm EP baseline;
• ORT MIGraphX EP;
• ORT MIGraphX EP with fp16 artifacts;
• ORT ROCm/MIGraphX plus CK FlashAttention custom-op artifacts, if compatible;
• CPU fallback baseline for correctness and degraded-mode behavior.

Metrics:

• model/session load and cold compile time;
• warm P50/P95/P99 latency;
• throughput under concurrency;
• peak and steady-state VRAM;
• correctness/logit drift versus PyTorch or existing ONNX baseline;
• fallback reason and provider ownership diagnostics.

Acceptance criteria

• A provider strategy doc or design note explains the AMD ONNX Runtime provider order and fallback policy.
• Classifier, token-classifier, embedding, and multimodal ONNX paths use a consistent provider-selection abstraction or clearly documented exceptions.
• The ROCm Docker/runtime build choice (rocm-dynamic vs migraphx-dynamic) is explicitly decided and validated.
• Benchmark results cover the router-owned signal models listed above and include both latency and memory data.
• Any MIGraphX correctness, operator coverage, custom-op, or cache limitations are captured as follow-up issues.
• Docs explain how operators can verify which provider was selected and why fallback happened.

Validation

make agent-report ENV=amd CHANGED_FILES="onnx-binding src/vllm-sr tools/docker tools/make e2e/config/onnx-binding src/semantic-router/pkg/classification src/semantic-router/pkg/embedding src/semantic-router/pkg/extproc src/fleet-sim"
make agent-lint CHANGED_FILES="onnx-binding src/vllm-sr tools/docker tools/make e2e/config/onnx-binding src/semantic-router/pkg/classification src/semantic-router/pkg/embedding src/semantic-router/pkg/extproc src/fleet-sim"
make agent-feature-gate ENV=amd CHANGED_FILES="onnx-binding src/vllm-sr tools/docker tools/make e2e/config/onnx-binding src/semantic-router/pkg/classification src/semantic-router/pkg/embedding src/semantic-router/pkg/extproc src/fleet-sim"

References

• ONNX Runtime ROCm EP notice: ROCm EP has been removed starting in ORT 1.23 and applications should migrate to MIGraphX EP.
• ONNX Runtime MIGraphX EP: MIGraphX accelerates ONNX models on AMD GPUs through AMD's graph optimization engine.
• AMD MIGraphX docs: performance depends on graph compilation, kernel fusion, tuning/cache behavior, and workload-specific validation.