[RFC][calling for help from the community]: Large-Scale Multi-Stage Serving Architecture for vLLM-Omni

Motivation.

This RFC proposes a stage-based large-scale serving architecture for vLLM-Omni.

The core idea is to treat the stage as the fundamental deployment unit: each stage can be launched independently via CLI, and each stage can scale with its own replica count. This enables flexible deployment topologies such as 1:1:1, 1:N:1, 1:N:M, and 1:N, where the ratio represents per-stage replica counts.

The primary goal is to support current multi-stage Omni and Diffusion models with a minimal yet extensible serving architecture covering:

• Multi-stage CLI deployment
• Stage-level instance discovery
• Coordinator-based instance state synchronization
• Stage-aware routing in the orchestrator
• Stage-ratio deployments across single-node and multi-node setups

Background

vLLM-Omni increasingly serves models that are naturally expressed as multi-stage pipelines rather than a single monolithic engine.

Representative examples include:

• Thinker → Talker → Code2Wav
• Thinker → ImageGen
• AR → DiT

In practice, different stages exhibit distinct throughput, latency, and memory characteristics, making a fixed 1:1 deployment insufficient. Independent stage scaling is required to match real workload profiles — for example:

• One Thinker paired with multiple Talker replicas
• One Thinker paired with multiple Talker and Code2Wav replicas
• One AR paired with multiple DiT replicas

Throughout this RFC, ratios such as 1:N:M denote different replica counts for different stages.

Goals

This RFC aims to:

1. Define stage as the first-class deployment unit for multi-stage serving
2. Support independent startup of each stage via CLI
3. Support multiple replicas per stage
4. Support stage-ratio deployments for current Omni and Diffusion pipelines
5. Provide a minimal serving architecture for registration, discovery, routing, and dispatch

Use Cases

Diffusion

HunyuanImage-3.0

Target topologies:

Topology	Scope	Parallelism	GPU Layout
AR:DiT → 1:1	Single node	TP4	4 + 4
AR:DiT → 1:3	Multi node	TP4	4 + 12
AR:DiT → 1:N	Multi node	—	—

Omni

Qwen3.5-Omni

Target topologies:

Topology	Scope	Notes
Thinker:Talker:Code2Wav → 1:1:1	Single node	Baseline
Thinker:Talker:Code2Wav → 1:N:1	Single node	Talker scale-out
Thinker:Talker:Code2Wav → 1:N:M	Single node	Talker + Code2Wav scale-out
Thinker:Talker:Code2Wav → x:y:z + Thinker PD disaggregation	Multi node	Full disaggregation

Ming-Flash-Omni-2.0

Target topologies:

Topology	Scope	Parallelism
Thinker:Talker → 1:1	Single node	Thinker TP4
Thinker:Talker → 1:4	Single node	Thinker TP4
Thinker:ImageGen → 1:1	Single node	Thinker TP4
Thinker:ImageGen → 1:4	Single node	Thinker TP4

High-Level Architecture

                    ┌──────────────────────┐
                    │      API Server       │
                    └──────────┬────────────┘
                               │
                               ▼
                    ┌──────────────────────┐
                    │     Orchestrator      │
                    └──────────┬────────────┘
                               │
                               ▼
           ┌───────────────────────────────────────────┐
           │      Omni Coordinator (Control Plane)     │
           │                                           │
           │  · Stage registration & deregistration    │
           │  · Heartbeat & liveness tracking          │
           │  · Stage instance discovery               │
           │  · Metrics collection & instance updates  │
           └────────────────┬──────────────────────────┘
                            │
                            ▼
             ┌──────────────────────────────────┐
             │        Stage Worker Pools        │
             │                                  │
             │  Thinker / Talker / Code2Wav     │
             │  AR / DiT / ImageGen / ...       │
             └──────────────────────────────────┘

Requirements

Functional

• CLI: Support launching multiple stage instances independently via command line
• Instance Discovery: Support service discovery organized by stage type
• Coordinator: Support synchronization of multi-stage instance state and metrics
• Orchestrator: Support routing requests across multiple replicas of each stage
• DiffusionEngineCore Decoupling: Support deployment of DiffusionEngineCore decoupled from EngineClient
• API Server: Support multiple API server instances

Performance

• Entrypoint profiling and optimization for Qwen3-Omni, HunyuanImage-3.0, and Ming-Flash-Omni-2.0
• Connector profiling and optimization for Qwen3-Omni, HunyuanImage-3.0, and Ming-Flash-Omni-2.0

RAS (Reliability, Availability, Serviceability)

• Instance health checks via heartbeat
• Automatic unhealthy instance isolation
• Dynamic instance join / leave without full-system restart

Design

1. CLI

Each stage must be independently launchable from the CLI.

The CLI should make stage-based deployment explicit, allowing users to start a single instance of one stage, multiple replicas of the same stage, or different stages with different replica counts. This is the foundation for 1:N:1, 1:N:M, and 1:N deployment topologies.

2. Instance Discovery

The serving stack requires stage-level instance discovery so that independently launched stage workers can be located and routed to correctly.

Discovery should be organized by stage type and expose the current live instance pool for each stage. This decouples the launch order and lifecycle of individual stage workers from the rest of the system.

3. Coordinator

The Coordinator serves as the control-plane component for multi-stage serving. Its responsibilities include stage instance registration, heartbeat-based liveness tracking, synchronization of instance state and metrics, maintenance of per-stage instance pools, and publishing instance updates to routing components.

This provides the minimal control-plane support needed for stage-ratio deployments without introducing heavyweight external dependencies.

4. Orchestrator

The Orchestrator executes requests over the stage graph. Its responsibilities include identifying the stage sequence for a given request, selecting an instance from the pool of each stage, routing across multiple replicas of a stage, and connecting independently scaled stage pools into one logical pipeline.

5. Load Balancing

Because each stage may have multiple replicas, the system needs stage-local load balancing. The initial requirement is balanced dispatch across the currently available instances of a stage. More advanced policies (e.g., latency-aware or queue-depth-aware routing) can be introduced later and are not in scope for this RFC.

6. DiffusionEngineCore / EngineClient Decoupling

For Diffusion pipelines, DiffusionEngineCore must support deployment in a separate process from EngineClient. This allows more flexible deployment layouts and is essential for multi-node Diffusion serving where the AR and DiT stages reside on different machines.

Roadmap

P0 — Critical Path

• Support Qwen3-Omni single-node 1:M:N deployment
• Support HunyuanImage-3.0 multi-node 1:N deployment with TP4

P1 — High Priority

• Support Qwen3-Omni Thinker PD-disaggregated multi-node deployment
• Expand model coverage to additional Omni and Diffusion models
• Support combinations with additional parallelism modes

P2 — Medium Priority

• Support multi-API-server deployment
• Implement instance health checks and unhealthy-instance isolation
• Complete entrypoint profiling and optimization
• Integrate high-performance connector support

P3 — Future

• Support dynamic instance join / leave at runtime

Validation Plan

Baseline

• Qwen3-Omni: Thinker:Talker:Code2Wav → 1:1:1 — single node
• HunyuanImage-3.0: AR:DiT → 1:1 — single node, TP4, 4+4 GPUs

April

• Qwen3-Omni: Thinker:Talker:Code2Wav → 1:N:1 — single node
• Qwen3-Omni: Thinker:Talker:Code2Wav → 1:N:N — single node
• HunyuanImage-3.0: AR:DiT → 1:3 — multi node, TP4, 4+12 GPUs

May

• Qwen3-Omni: Thinker:Talker:Code2Wav → x:y:z + Thinker PD disaggregation — multi node

Expected Outcome

After this work, vLLM-Omni will support a serving model where each stage can be launched independently, each stage can have multiple replicas, different stages can use different replica counts, and the serving stack can route requests across these stage pools as a single logical pipeline.

This is the foundational architecture required to support practical, large-scale, multi-stage serving in vLLM-Omni.

Proposed Change.

Please provide the detailed design document of the RFC using the template.

Feedback Period.

No response

CC List.

No response

Any Other Things.

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.