Motivation.

RFC: Data Parallelism for Video Generation Models in vLLM-Omni

Abstract

This RFC proposes a comprehensive, model-agnostic Data Parallelism (DP) extension for vLLM-Omni targeting video generation models, building upon the foundational work in PR #4460 ("Omni-Replica + vLLM-DP/EP by YAML"). PR #4460 introduces YAML-based DP/EP configuration for the AR stage, fixes critical KV-transfer port collision bugs, and enables multi-replica deployments with inner-DP support. This RFC extends that foundation to video generation models (e.g., HunyuanVideo 1.5, 8.3B), providing a generic DP system that enables linear throughput scaling across NVIDIA H100 and Ascend 910B platforms.

---

1. Motivation

1.1 The Throughput Challenge

Video generation models are growing rapidly in size. Production deployments with high request concurrency require throughput scaling beyond what a single device can provide. Data Parallelism (DP) addresses this by replicating the model across multiple devices and processing independent requests in parallel.

Model	Parameters	Architecture
HunyuanVideo 1.5	8.3B	Diffusion Transformer (DiT)

1.2 Current State in vLLM-Omni

vLLM-Omni already supports multiple parallelism strategies:

Strategy	Primary Use Case	Throughput Gain
Tensor Parallelism (TP)	Large model memory reduction	Low-Medium
Sequence Parallelism (SP)	Long sequences	High
Pipeline Parallelism (PP)	Multi-stage pipelines	Medium
Data Parallelism (DP)	High request throughput	Linear

vLLM-Omni provides foundational DP infrastructure:

• _DIT GroupCoordinator for DiT-specific DP groups
• DP rank tracking in GPUARWorker.init_device()
• data_parallel_backend configuration supporting "mp" and "ray" backends
• PR #4460: YAML-based DP/EP configuration, multi-replica support, and KV-transfer fixes

1.3 PR #4460 Foundation

PR #4460 establishes the core infrastructure that this RFC builds upon:

PR #4460 Feature	Description	Relevance to This RFC
YAML-based DP/EP	Support vLLM DP and EP by YAML config for AR stage	Enables declarative DP configuration
Multi-replica DP	Inner-DP workers spread across devices per replica	Enables per-replica independent processing
KV-transfer DP fix	ZMQ port includes DP rank to avoid EADDRINUSE	Enables multi-rank KV cache management
Device count fix	Per-stage device count includes data_parallel_size	Correct device allocation for DP
DP coordinator	DPCoordinator for inner-DP engine-core spawning	DP group management for video models
CLI deprecation	--omni-dp-size-local → --omni-num-replica	Clean DP configuration interface

1.4 Design Goals

1. Build on PR #4460: Leverage existing YAML-based DP/EP infrastructure
2. Model-agnostic: Works with any registered video generation model
3. Linear throughput scaling: Near-linear speedup with DP size
4. Hardware-agnostic: NVIDIA H100 and Ascend 910B
5. Composable: Works alongside TP, SP, and CFG-Parallel
6. Production-ready: Load balancing, KV-transfer, and monitoring

---

2. Target Model

Through vLLM-Omni's model registry, DP supports any registered video generation model. This RFC primarily targets HunyuanVideo 1.5 as the reference model:

Model	Parameters	Registry Key
HunyuanVideo 1.5	8.3B	HunyuanVideo-1.5

Additional models are automatically supported once registered in vLLM-Omni, with no code changes required.

---

3. Design

3.1 Architecture Overview

The architecture builds on PR #4460's multi-replica DP infrastructure:

┌─────────────────────────────────────────────────────────────────────────────┐
│              Data Parallel Architecture for Video Models                   │
│                    (Based on PR #4460 Infrastructure)                     │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                             │
│  ┌─────────────────────────────────────────────────────────────────────┐   │
│  │                     YAML Configuration                              │   │
│  │  data_parallel_size: 4, tensor_parallel_size: 1, replicas: 2       │   │
│  └─────────────────────────────────────────────────────────────────────┘   │
│                                    │                                       │
│                                    ▼                                       │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐   │
│  │   Rank 0     │  │   Rank 1     │  │   Rank 2     │  │   Rank 3     │   │
│  │   Model      │  │   Model      │  │   Model      │  │   Model      │   │
│  │   (Full)     │  │   (Full)     │  │   (Full)     │  │   (Full)     │   │
│  └──────┬───────┘  └──────┬───────┘  └──────┬───────┘  └──────┬───────┘   │
│         │                 │                 │                 │           │
│    ┌────┴────┐       ┌────┴────┐       ┌────┴────┐       ┌────┴────┐   │
│    │Request  │       │Request  │       │Request  │       │Request  │   │
│    │Batch A  │       │Batch B  │       │Batch C  │       │Batch D  │   │
│    └─────────┘       └─────────┘       └─────────┘       └─────────┘   │
│         │                 │                 │                 │           │
│         └─────────────────┴─────────────────┴─────────────────┘           │
│                                    │                                       │
│                                    ▼                                       │
│                    ┌───────────────────────────────┐                       │
│                    │   DPCoordinator (PR #4460)    │                       │
│                    │   Inner-DP engine spawning    │                       │
│                    └───────────────────────────────┘                       │
│                                                                             │
│  Communication Pattern: None between ranks (independent replicas)          │
│  Throughput Scaling: Linear with DP size                                   │
│                                                                             │
└─────────────────────────────────────────────────────────────────────────────┘

3.2 YAML-Based Configuration (PR #4460)

Following PR #4460's approach, DP is configured via YAML:

# configs/hunyuanvideo_dp.yaml
model: HunyuanVideo-1.5
omni: true

# PR #4460: DP/EP configuration
data_parallel_size: 4
tensor_parallel_size: 1
enable_expert_parallel: false  # For MoE models

# PR #4460: Multi-replica support
omni_num_replica: 2  # Number of replicas per stage

# PR #4460: KV-transfer configuration
connectors:
  kv_connector:
    name: MooncakeTransferEngineConnector
    extra:
      host: "auto"
      zmq_port: 50051
      protocol: "tcp"

# Video generation parameters
num_frames: 49
height: 480
width: 720
num_inference_steps: 50
dtype: fp16

3.3 DP Worker

class VideoDPWorker:
    """
    Generic DP worker: each rank loads a full model copy independently.
    Leverages PR #4460's DPCoordinator for inner-DP engine spawning.
    """

    def __init__(self, config: DPConfig):
        self.rank = config.rank
        self.local_rank = config.local_rank
        self.device_type = config.device_type  # "cuda" or "npu"
        self.device = torch.device(f"{self.device_type}:{self.local_rank}")

        # PR #4460: Use DPCoordinator for inner-DP management
        self.dp_coordinator = DPCoordinator(
            dp_size=config.data_parallel_size,
            replica_id=config.replica_id
        )

        # Load model - type determined by config
        self.model = self._load_model(config.model_config)
        self.model.to(self.device)

    def _load_model(self, model_config):
        """
        Load model via vLLM-Omni's model registry.
        PR #4460: Uses per-stage device count including data_parallel_size
        """
        from vllm_omni.diffusion import load_diffusion_model

        return load_diffusion_model(
            model_name=model_config.model_name,
            tensor_parallel_size=1,  # DP internally uses TP=1
            dtype=model_config.dtype
        )

    def infer(self, requests: List[Request]) -> List[GeneratedVideo]:
        """
        Execute inference on assigned requests.
        PR #4460: Each replica gets independent DP group
        """
        outputs = []
        for req in requests:
            if req.type == "text_to_video":
                video = self.model.generate(
                    prompt=req.prompt,
                    num_frames=req.num_frames,
                    height=req.height,
                    width=req.width
                )
            elif req.type == "image_to_video":
                video = self.model.generate(
                    image=req.image,
                    prompt=req.prompt,
                    num_frames=req.num_frames,
                    height=req.height,
                    width=req.width
                )
            else:
                raise ValueError(f"Unsupported request type: {req.type}")
            outputs.append(video)
        return outputs

3.4 KV-Transfer with DP-Aware Ports (PR #4460)

PR #4460 fixes KV-transfer port collisions when inner-DP > 1:

class DPWareKVTransferManager:
    """
    PR #4460: KV-transfer with DP-aware ZMQ ports.
    Fixes EADDRINUSE when KV-sending stage uses inner data_parallel_size > 1
    """

    def __init__(self, config: KVTransferConfig):
        self.dp_size = config.data_parallel_size
        self.dp_rank = config.data_parallel_rank

        # PR #4460: Port includes dp_rank to avoid collisions
        self.zmq_port = config.base_port + (
            config.replica_id * KV_DP_PORT_STRIDE +
            config.dp_rank * KV_DP_PORT_STRIDE +
            config.tp_rank
        )

    def get_sender_info(self, dp_rank: int = 0):
        """
        PR #4460: get_sender_info includes dp_rank parameter
        """
        return {
            "zmq_port": self.zmq_port + dp_rank * KV_DP_PORT_STRIDE,
            "dp_rank": dp_rank,
        }

3.5 DP Scheduler

class VideoDPScheduler:
    """
    Generic Data Parallel scheduler for video generation models.
    Model-agnostic: works with any registered diffusion model.
    """

    def __init__(self, config: DPConfig):
        self.dp_size = config.data_parallel_size
        self.rank = config.rank
        self.load_balancer = self._create_load_balancer(config)

    def schedule(self, requests: List[Request]) -> List[Request]:
        """Distribute requests to the current rank."""
        if self.config.load_balancing == "round_robin":
            return self._round_robin(requests)
        else:
            return self._dynamic(requests)

    def _round_robin(self, requests):
        """Round-robin: request i → rank i % dp_size"""
        return [req for i, req in enumerate(requests)
                if i % self.dp_size == self.rank]

    def _dynamic(self, requests):
        """Dynamic load balancing based on estimated cost."""
        return self.load_balancer.distribute(requests, self.rank)

3.6 Recommended Hardware Configurations

Hardware Platform	Model	Memory	Per-Rank Throughput	Recommended DP Size
NVIDIA GPU	H100	80GB HBM3	2–3 concurrent requests	4–8
Ascend NPU	910B	64GB HBM2e	1–2 concurrent requests	4–8

---

4. Configuration

4.1 Configuration Parameters

@dataclass
class VideoDPConfig:
    # Core DP settings (PR #4460 aligned)
    enable_dp: bool = False
    data_parallel_size: int = 1
    omni_num_replica: int = 1  # PR #4460: replaces omni_dp_size_local

    # Deployment (PR #4460)
    data_parallel_backend: str = "mp"  # "mp" | "ray"
    nnodes_within_dp: int = 1
    device_type: str = "cuda"  # "cuda" | "npu"

    # Load balancing
    load_balancing: str = "round_robin"  # "round_robin" | "dynamic"

    # Model configuration
    model_name: str = None  # From model registry
    model_config: dict = None
    dtype: str = "fp16"
    tensor_parallel_size: int = 1

    # Generation parameters
    num_frames: int = 49
    height: int = 480
    width: int = 720
    num_inference_steps: int = 50

4.2 Command-Line Usage

NVIDIA H100

# PR #4460: YAML-based deployment
vllm serve --config configs/hunyuanvideo_dp.yaml

# Or CLI (with PR #4460 --omni-num-replica)
vllm serve HunyuanVideo-1.5 --omni \
    --data-parallel-size 4 \
    --omni-num-replica 2 \
    --load-balancing dynamic

# DP + TP combination
vllm serve HunyuanVideo-1.5 --omni \
    --data-parallel-size 4 \
    --tensor-parallel-size 2 \
    --omni-num-replica 2

Ascend 910B

# 4× Ascend 910B
vllm serve HunyuanVideo-1.5 --omni \
    --data-parallel-size 4 \
    --omni-num-replica 2 \
    --load-balancing dynamic \
    --device npu

# 8× Ascend 910B
vllm serve HunyuanVideo-1.5 --omni \
    --data-parallel-size 8 \
    --omni-num-replica 4 \
    --device npu

4.3 Multi-Node Deployment (PR #4460)

PR #4460 supports multi-host deployments with head + headless runtimes:

# Head node (stage 0)
vllm serve --config configs/hunyuanvideo_dp.yaml --stage-id 0

# Headless node (stage 1)
vllm serve --config configs/hunyuanvideo_dp.yaml --headless --stage-id 1

---

5. Performance Expectations

5.1 Throughput Scaling

DP Size	Theoretical	Expected	Efficiency
1	1.0×	1.0×	100%
2	2.0×	1.96×	98%
4	4.0×	3.84×	96%
8	8.0×	7.6×	95%

5.2 Memory Footprint

Model	Hardware	Configuration	Per Rank Memory
HunyuanVideo 1.5 (8.3B)	H100	FP16	~16.6 GB
HunyuanVideo 1.5 (8.3B)	Ascend 910B	INT8	~8.3 GB

5.3 Hardware-Specific Performance

Hardware	Single Request Latency	Max Concurrent Per Rank
H100	Model-dependent	2–3 requests
Ascend 910B	Model-dependent	1–2 requests

---

6. Composability with Other Parallel Strategies

Combination	Compatibility	Configuration
DP + TP	✅	data_parallel_size: N, tensor_parallel_size: M
DP + SP	✅	data_parallel_size: N, ulysses_degree: M
DP + CFG	✅	data_parallel_size: N, cfg_parallel_size: M
DP + EP (MoE)	✅	data_parallel_size: N, enable_expert_parallel: true

---

7. Implementation Roadmap

Phase	Tasks	Dependencies
Phase 1	YAML DP configuration for video models	PR #4460 YAML infra
Phase 2	Multi-rank independent model loading	PR #4460 DPCoordinator
Phase 3	Dynamic load balancing	Phase 2
Phase 4	KV-transfer with DP-aware ports	PR #4460 KV fixes
Phase 5	Multi-node deployment	PR #4460 head/headless
Phase 6	Performance tuning & validation	Phase 5

---

8. Validation Checklist

• YAML-based DP configuration works (PR #4460 style)
• Each DP rank loads the specified model independently
• Each rank processes different T2V/I2V requests
• Output correctness matches single-rank baseline
• Throughput scales linearly with DP size (1, 2, 4, 8)
• Round-robin and dynamic load balancing work correctly
• Multi-node deployment (2 nodes × 4 ranks)
• DP + TP combination works
• DP-aware KV-transfer ports work (no EADDRINUSE)
• H100 (CUDA) and Ascend 910B (CANN) support
• --omni-num-replica works (PR #4460 CLI)

---

9. References

• PR #4460: Omni-Replica + vLLM-DP/EP by YAML
• vLLM-Omni Parallelism Strategies
• vLLM-Omni Diffusion Model Registry
• HunyuanVideo 1.5 Technical Report
• vLLM-Omni _DIT GroupCoordinator

---

Proposed Change.

Please provide the detailed design document of the RFC using the template.

Feedback Period.

No response provided.

CC List.

@hsliuustc0106 @Gaohan123 @david6666666 @bjf-frz

Any Other Things.

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.