feat: add TensorRT-LLM and Triton TensorRT-LLM inference-aware backend adapter

Summary

Add a TensorRT-LLM / Triton TensorRT-LLM adapter for inference-aware backend routing once the engine-neutral backend telemetry and replica identity contract is defined.

Parent roadmap: #2287 Parent issue: #2332 Depends on: #2349 Related: #2350, #2351, #2352, #2353, #2359, #2372, #2376 Related external docs: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/getting_started/trtllm_user_guide.html, https://nvidia.github.io/TensorRT-LLM/advanced/kv-cache-reuse.html, https://nvidia.github.io/TensorRT-LLM/blogs/tech_blog/blog5_Disaggregated_Serving_in_TensorRT-LLM.html Related code: src/semantic-router/pkg/config, src/semantic-router/pkg/selection, src/semantic-router/pkg/latency, src/semantic-router/pkg/inflight, src/semantic-router/pkg/extproc, src/vllm-sr/cli/templates/envoy.template.yaml, deploy/operator, deploy/kubernetes, deploy/recipes, src/fleet-sim, observability and replay diagnostics

Motivation

TensorRT-LLM is a first-class NVIDIA inference engine for high-performance LLM serving. It exposes engine concepts that matter for backend-aware routing, including scheduling behavior, KV cache reuse, and disaggregated serving patterns. Triton TensorRT-LLM deployments are also a common production serving shape for NVIDIA GPU clusters.

The router should support TensorRT-LLM through the same normalized backend telemetry contract used by vLLM, SGLang, and ATOM. TensorRT-LLM details should not leak into logical model selection, decision evaluation, or provider config.

This issue is intentionally different from a future NVIDIA Dynamo integration. Dynamo is a serving/orchestration layer above engines such as TensorRT-LLM, vLLM, and SGLang. This issue is for a direct TensorRT-LLM / Triton TensorRT-LLM engine adapter.

Scope

Define and implement, or first design if telemetry surfaces need confirmation, TensorRT-LLM support for:

• backend and replica identity mapping for direct TensorRT-LLM and Triton TensorRT-LLM deployments;
• endpoint discovery and config representation using canonical providers.models[].backend_refs[];
• telemetry ingestion for health, queue depth or scheduler backlog, active requests, TTFT/TPOT/E2E latency, GPU/memory pressure, and Triton/TensorRT-LLM metrics where available;
• KV cache reuse, block reuse, prefix/cache affinity, and disaggregated prefill/decode signals where exposed by the deployment;
• stale or missing telemetry fallback to existing model-level routing behavior;
• diagnostics that distinguish TensorRT-LLM telemetry freshness, selected backend/replica, and backend policy reason;
• examples or docs for direct TensorRT-LLM, Triton TensorRT-LLM, and how this differs from Dynamo-managed serving.

Non-goals

• Do not implement NVIDIA Dynamo integration here; Dynamo should be tracked as a serving-stack/orchestration integration.
• Do not replace TensorRT-LLM or Triton internal scheduling.
• Do not require NVIDIA hardware or TensorRT-LLM in default CPU CI.
• Do not add TensorRT-LLM-specific fields to core backend policy when normalized telemetry fields are sufficient.
• Do not block vLLM/SGLang/ATOM adapters on TensorRT-LLM availability.

Acceptance criteria

• TensorRT-LLM and Triton TensorRT-LLM backends can be represented as backend-aware candidates under one logical model.
• TensorRT-LLM telemetry maps to the normalized backend telemetry contract from #2349.
• The adapter has explicit freshness/TTL and fail-open behavior for missing or stale telemetry.
• Backend policy diagnostics can explain selected logical model, selected TensorRT-LLM backend/replica when available, telemetry freshness, and policy reason.
• Direct TensorRT-LLM/Triton TensorRT-LLM integration is documented separately from NVIDIA Dynamo integration.
• Tests or documented validation cover config parsing, telemetry normalization, stale telemetry fallback, and replay/diagnostic output without requiring TensorRT-LLM in default CPU CI.

Validation

make agent-report ENV=cpu CHANGED_FILES="src/semantic-router/pkg/config src/semantic-router/pkg/selection src/semantic-router/pkg/latency src/semantic-router/pkg/inflight src/semantic-router/pkg/extproc src/vllm-sr/cli/templates deploy/operator deploy/kubernetes deploy/recipes src/fleet-sim website/docs"
make agent-lint CHANGED_FILES="src/semantic-router/pkg/config src/semantic-router/pkg/selection src/semantic-router/pkg/latency src/semantic-router/pkg/inflight src/semantic-router/pkg/extproc src/vllm-sr/cli/templates deploy/operator deploy/kubernetes deploy/recipes src/fleet-sim website/docs"
make agent-ci-gate CHANGED_FILES="src/semantic-router/pkg/config src/semantic-router/pkg/selection src/semantic-router/pkg/latency src/semantic-router/pkg/inflight src/semantic-router/pkg/extproc src/vllm-sr/cli/templates deploy/operator deploy/kubernetes deploy/recipes src/fleet-sim website/docs"