vllm-project/semantic-router

feat: add ROCm ATOM inference-aware backend adapter

Open

#2,352 opened on Jul 5, 2026

View on GitHub
 (0 comments) (0 reactions) (1 assignee)Go (699 forks)github user discovery
area/model-selectionarea/observabilityenhancementenvironment/rocmhelp wantedoperationspriority/P2roadmap

Repository metrics

Stars
 (4,293 stars)
PR merge metrics
 (PR metrics pending)

Description

Summary

Add a ROCm ATOM adapter for inference-aware backend routing so AMD self-hosted deployments can expose ATOM runtime, cache, and profiling signals through the same backend policy used by other engines.

Parent roadmap: #2287 Parent issue: #2332 Depends on: engine-neutral backend telemetry contract Related external project: https://github.com/ROCm/ATOM Related code: src/semantic-router/pkg/config, src/semantic-router/pkg/selection, src/semantic-router/pkg/latency, deploy/amd, deploy/operator, src/fleet-sim

Motivation

ATOM is a ROCm-focused lightweight vLLM-like implementation with OpenAI-compatible serving, prefix caching, P/D disaggregation, profiling, and benchmark surfaces. Those capabilities are directly relevant to self-hosted AMD routing, but they should enter vLLM Semantic Router through a normalized engine adapter rather than through ATOM-specific routing branches.

Scope

Define and implement ATOM support for:

  • ATOM backend/replica identity under a logical model;
  • OpenAI-compatible serving endpoint integration;
  • queue, latency, GPU, health, prefix-cache, and disaggregation-related telemetry where available;
  • ROCm/AMD deployment examples under deploy/amd or related Kubernetes profiles;
  • diagnostics that distinguish ATOM telemetry freshness and fallback from vLLM/SGLang.

Non-goals

  • Do not require ATOM for default CPU CI.
  • Do not duplicate ATOM's benchmark dashboard inside the router.
  • Do not make AMD-specific telemetry fields part of the generic backend policy unless they are normalized.

Acceptance criteria

  • ATOM backends can be modeled as backend-aware candidates without changing logical recipe/decision behavior.
  • ATOM telemetry maps to the normalized backend telemetry contract.
  • Missing telemetry falls back to existing routing behavior.
  • AMD/ATOM examples document opt-in setup and validation expectations.

Contributor guide