perf: analyze and optimize ROCm production performance for high concurrency and long context

Summary

Perform a deep ROCm production performance analysis for vLLM Semantic Router so self-hosted AMD GPU deployments can handle high concurrency, long context, and multiple router-owned models while using GPUs such as MI300X and MI355X effectively.

Parent roadmap: #2287 Related: #1510, #2332, #2352, #2359 Related code: deploy/amd, .github/workflows/docker-release.yml, src/vllm-sr/cli, src/vllm-sr/Dockerfile.rocm, tools/docker/Dockerfile.extproc-rocm, perf, src/fleet-sim, src/semantic-router/pkg/selection, src/semantic-router/pkg/modelruntime, src/semantic-router/pkg/extproc, src/semantic-router/pkg/observability

Motivation

The repo already has an AMD ROCm profile, ROCm container images, vllm-sr serve --platform amd, component performance tests, and fleet-sim capacity modeling. H2 needs a deeper production-oriented ROCm analysis that looks beyond functional deployment:

• high-concurrency request routing;
• long-context requests and KV/cache pressure;
• multiple router model assets serving at the same time;
• classifier/embedding/modelruntime overhead;
• interaction with vLLM/ATOM-style ROCm serving backends;
• GPU utilization and memory pressure on MI300X, MI355X, and similar AMD accelerators.

Scope

Analyze and propose optimizations for:

• router request-path CPU/GPU overhead under high concurrency;
• long-context signal extraction, context routing, cache affinity, and replay overhead;
• memory pressure and model loading when multiple router-owned models are served concurrently;
• ROCm image/runtime configuration, including AMD GPU passthrough and AITER/vLLM/ATOM integration assumptions;
• observability needed to attribute bottlenecks to router, inference engine, GPU, cache, storage, or Envoy;
• fleet-sim/perf harness coverage for MI300X/MI355X-like capacity planning.

Non-goals

• Do not replace vLLM, ATOM, or ROCm engine-level profiling.
• Do not hard-code MI300X/MI355X-only behavior into generic router code.
• Do not treat one benchmark run as sufficient production proof.

Acceptance criteria

• A ROCm production performance report identifies bottlenecks, measurement method, hardware profile, and workload mix.
• The report includes high-concurrency and long-context scenarios.
• The analysis covers multiple router-owned model assets served concurrently.
• Proposed optimizations are split into follow-up issues or PRs with clear owner surfaces.
• Perf/fleet-sim coverage is extended or documented for AMD GPU capacity planning.
• Docs explain recommended ROCm deployment and observability settings for production validation.

Validation

make agent-report ENV=amd CHANGED_FILES="deploy/amd src/vllm-sr perf src/fleet-sim src/semantic-router/pkg/selection src/semantic-router/pkg/modelruntime src/semantic-router/pkg/extproc src/semantic-router/pkg/observability website/docs .github/workflows"
make agent-lint CHANGED_FILES="deploy/amd src/vllm-sr perf src/fleet-sim src/semantic-router/pkg/selection src/semantic-router/pkg/modelruntime src/semantic-router/pkg/extproc src/semantic-router/pkg/observability website/docs .github/workflows"
make agent-feature-gate ENV=amd CHANGED_FILES="deploy/amd src/vllm-sr perf src/fleet-sim src/semantic-router/pkg/selection src/semantic-router/pkg/modelruntime src/semantic-router/pkg/extproc src/semantic-router/pkg/observability website/docs .github/workflows"

References

• AMD Ryzen/ROCm and AI PC specs should be verified from official AMD sources when edge or client hardware is involved.
• ATOM adapter work is tracked in #2352.