[RFC]: Pipeline Parallelism & Stream Batch for Real-Time Video Generation

1. Motivation

1.1 Why Sequence Parallelism Fails for Streaming

vLLM-Omni's diffusion parallelism relies on Sequence Parallelism (Ulysses-SP, Ring Attention). This works well for offline generation with large chunks (~33,000+ tokens), where communication overhead is amortized over heavy compute.

Live-streaming operates in a fundamentally different regime. A causal DiT generating 4 frames at 480p produces only ~1,536 tokens per forward pass — far below the H100 roofline ridge point (590.75 FLOP/Byte), placing the workload deeply memory-bound. In this regime:

• SP's all-to-all adds 40–120ms per forward pass on NVLink H100, 20–40× higher than block-level PP's inter-stage P2P cost (~2–6ms)
• Even with perfect compute scaling, SP cannot satisfy sub-second TTFF SLOs for a 14B model
• The StreamDiffusionV2 paper (arXiv:2511.07399, MLSys 2026) measures this directly: SP yields essentially no speedup at 1,536 tokens, while block-level PP achieves near-linear FPS scaling

This is a workload regime mismatch, not a tuning problem.

1.2 Evidence: StreamDiffusionV2 Results

Key results on 4×H100 (NVLink), without TensorRT or quantization:

Model	Resolution	Steps	FPS	TTFF
Wan2.1-1.3B	480p	1	42.26	0.37s
Wan2.1-14B	480p	4	58.28	<0.5s

Counter-intuitively, more denoising steps → higher FPS: deeper in-flight pipelines increase GPU utilization (Stream Batch effect, §2.2). On 4×RTX 4090 (PCIe), ~16 FPS is still achievable — PP remains effective over PCIe where SP completely fails.

1.3 Connection to World Model Infrastructure

RFC #1987 (World Model Support) requires autoregressive chunk diffusion, rolling KV cache, and multi-turn stateful sessions. All of these depend on low-latency multi-GPU execution. Pipeline PP & Stream Batch is the execution substrate that makes world model serving feasible at scale — it is an explicit P1 item in the Q2 roadmap (#2226 §3, "RFC needed").

1.4 Disambiguation: This RFC vs #647 (PipeFusion) and #268 (Async Stages)

Three separate "pipeline" concepts exist in vLLM-Omni:

	#647 PipeFusion	#268 Async Stages	This RFC
Axis	Spatial patches (sequence)	Cross-model stages (LLM→DiT→TTS)	Model depth (block layers)
Mechanism	Stale KV reuse across steps	Async hidden-state handoff	Stream Batch: steps as batch multiplier
Target	Offline high-res (>4096px)	Omni TTFT reduction	Real-time streaming (SLO <500ms TTFF)
Status	Open RFC, no PR	Phase 1 merged (#727)	RFC needed, Q2 open slot

Why PipeFusion (#647) does not solve the streaming problem — three independent reasons:

1. Warmup overhead cannot be amortized. PipeFusion requires a warmup step that degrades to serial execution. For 50-step offline generation, warmup costs ~2%. For 1–4 step streaming, it costs 25–100%, eliminating all parallel benefit. The xDiT paper explicitly states for FLUX.1 Schnell (4 steps): "Since the step number is very small, we do not apply PipeFusion."

2. Stale KV reuse breaks in causal generation. PipeFusion's savings assume high inter-step feature similarity. In causal autoregressive generation, each chunk's content shifts with user input — stale KV introduces semantic errors.

3. Empirical data from xDiT Performance Reports (arXiv:2411.01738, NeurIPS 2025): on 4×H100 NVLink with FLUX.1-dev, SP-Ulysses-4 achieves 1.63s vs PipeFusion which was not even benchmarked (paper: "Due to H100's excellent NVLink bandwidth, using USP is more appropriate than using PipeFusion"). On 8×A100 NVLink, PipeFusion is worse than SP-Ring at high resolution, and worse than Ulysses at 4096px. PipeFusion's advantage is narrow: PCIe + many steps + no skip connections.

Why #268 (Async Stages) is orthogonal: #268 handles async between model-level stages (LLM Stage → DiT Stage). This RFC handles parallelism inside the DiT Stage. Both can coexist: #268 manages the stage graph, this RFC manages intra-DiT execution.

Shared infrastructure with #647: The underlying P2P communication primitives (dist.isend, dual CUDA stream pattern) are identical. We propose a shared vllm_omni/diffusion/distributed/p2p_comm.py usable by both.

---

2. Design

The proposed system has two orthogonal parallelism dimensions.

2.1 Dimension 1: Pipeline Parallelism (Model Depth)

Partition DiT Transformer blocks along the depth dimension across GPUs:

Input frames
     │
     ▼
GPU 0: [VAE Encode] → broadcast noisy_latents → [DiT 0..k0]
                                                      │ isend
                                                      ▼
GPU 1:                                          [DiT k0..k1]
                                                      │ isend
                                                      ▼
GPU 2:                                          [DiT k1..k2]
                                                      │ isend
                                                      ▼
GPU 3:                                          [DiT k2..N] → [VAE Decode] → frames

Communication pattern (from StreamDiffusionV2 source): Sending uses dist.isend (non-blocking) on a dedicated com_stream; receiving uses dist.recv (blocking) on com_stream, synchronized to the compute stream via current_stream.wait_stream(com_stream). When the pipeline is fully loaded, the blocking recv for step K+1 is pre-satisfied by the isend from step K — forming natural overlap without explicit double-buffering.

VAE note: VAE encode runs only on rank 0 and is broadcast to all ranks (needed for cross-attention context). This broadcast is an optimization target (see §4).

2.2 Dimension 2: Stream Batch (Denoising Steps as Batch Multiplier)

With n denoising steps and n GPUs, the pipeline in steady state has n latents at different noise levels simultaneously in-flight:

micro-step →    t=1           t=2           t=3           t=4
GPU 0:       [noise_lvl 4]  [noise_lvl 3]  [noise_lvl 2]  [noise_lvl 1]
GPU 1:                      [noise_lvl 4]  [noise_lvl 3]  [noise_lvl 2]
GPU 2:                                     [noise_lvl 4]  [noise_lvl 3]
GPU 3:                                                    [noise_lvl 4] → clean frame

Every micro-step produces one clean output frame. More steps → higher FPS because the deeper pipeline keeps all GPUs busy. The SLO-aware scheduler dynamically adjusts batch size B and step count n to maximize GPU utilization while satisfying per-frame deadlines, using the StreamDiffusionV2 latency model: L(T,B) ≈ (A(T,B) + P_model) / (η × BW_HBM), giving f ∝ B/(1+B).

2.3 VAE Stage Balance

VAE encode/decode accounts for ~30% of total inference time and is concentrated on boundary ranks, causing pipeline imbalance. Two complementary solutions:

• Short-term (two-phase partition): At startup, compute_balanced_block_split assigns fewer DiT blocks to boundary ranks to compensate for VAE overhead. At runtime, BlockScheduler can optionally collect per-rank timing via dist.all_gather and rebalance — matching StreamDiffusionV2's _handle_block_scheduling pattern.
• Long-term (depends on #2089): Separate VAE as an independent stage co-located with the LLM Thinker, removing it from the DiT pipeline entirely.

---

3. Proposed Changes

3.1 New module: vllm_omni/diffusion/distributed/pp_pipeline.py

Placed in distributed/ alongside the existing cfg_parallel.py, sp_plan.py, etc. Following the existing Mixin convention (CFGParallelMixin, ProgressBarMixin), PP execution is exposed as a mixin that pipeline_*.py classes inherit — consistent with how Wan22Pipeline(nn.Module, CFGParallelMixin, ...) is structured today.

Key interfaces (implementation details in follow-up PRs):

class PipelineParallelMixin:
    """
    Mixin for pipeline_*.py classes. Provides PP-aware noise prediction
    and latent handoff methods, following the CFGParallelMixin pattern.
    Usage: class Wan22Pipeline(nn.Module, PipelineParallelMixin, CFGParallelMixin, ...)
    """
    def predict_noise_maybe_with_pp(self, positive_kwargs, negative_kwargs) -> torch.Tensor: ...
    def execute_micro_step(self, batch, kv_cache, outstanding) -> StageOutput: ...

class DiffusionPPExecutor:
    """Low-level P2P communication primitives. Used by PipelineParallelMixin."""
    com_stream: torch.cuda.Stream        # dedicated comm stream

    def send_activation_async(self, tensor, dst_rank, tag) -> dist.Work: ...
    def recv_activation(self, buffer, src_rank, tag) -> None: ...
    def broadcast_vae_output(self, tensor, src=0) -> None: ...

class StreamBatchScheduler:
    """
    SLO-aware scheduler managing n in-flight latents across denoising steps.
    Runs on rank 0; other ranks follow. Sits above PipelineParallelMixin.
    """
    def compute_optimal_config(self, observed_latency_ms, queue_depth) -> SchedulerConfig: ...
    def schedule_micro_step(self, in_flight, new_requests, config) -> MicroBatch: ...

def compute_balanced_block_split(
    num_blocks, num_stages,
    vae_encode_overhead=0.15, vae_decode_overhead=0.15,
) -> list[tuple[int, int]]: ...

class BlockScheduler:
    """Optional runtime rebalancing via all_gather timing."""
    def collect_timing_and_rebalance(self, local_dit_time_ms, partition) -> list[tuple[int, int]]: ...

class StreamVAE:
    """Chunk-wise VAE with 3D conv feature caching for temporal coherence."""
    def encode_chunk(self, frames) -> torch.Tensor: ...
    def decode_chunk(self, latents) -> torch.Tensor: ...

3.2 New module: vllm_omni/diffusion/distributed/p2p_comm.py

Shared P2P primitives (dist.isend/dist.recv + com_stream pattern) reusable by both this RFC and PipeFusion (#647). Placed in distributed/ alongside the existing group_coordinator.py which already contains PipelineGroupCoordinator.

---

4. Integration with Existing Infrastructure

4.1 Relation to RFC #874 (Step-wise Batching) — prerequisites already merged

RFC #874's core interfaces have already landed:

• #1368 (merged): prepare_encode / denoise_step / post_decode at the runner layer
• #1625 (merged, 2026-03-23): DiffusionRequestExecutionState, clean Scheduler/Executor separation

This RFC can begin immediately. denoise_step() from #1368 becomes execute_micro_step() within each pipeline stage. StreamBatchScheduler slots above the existing executor without invasive changes.

4.2 Relation to RFC #2089 (VAE as Separate Stage)

PipelineParallelMixin accepts an optional vae_stage parameter. If #2089 is available, VAE is delegated to the external stage; otherwise, StreamVAE runs inline. The short-term compute_balanced_block_split workaround means this RFC does not block on #2089.

4.3 Relation to RFC #1987 (World Model / Rolling KV Cache)

Each stage independently manages KV for its own DiT layers (verified in StreamDiffusionV2 source). Cross-stage coordination is minimal: only the rolling window boundary (a single int) needs broadcasting at chunk boundaries. This RFC provides a stub DistributedKVCache interface; full implementation is deferred to #1987.

4.4 Composability with Sparse Attention

Pipeline parallelism and sparse attention are orthogonal acceleration dimensions that compose multiplicatively. PipelineParallelMixin.execute_micro_step() calls the DiT blocks through the existing attention dispatch path — whether that resolves to dense or sparse is transparent to the PP layer. If the active attention backend changes at runtime, BlockScheduler should re-trigger its startup partition, as the effective FLOPs-per-block will have shifted.

4.5 Relation to #2363 (RFC) and #2322 (Draft PR) — Pipeline Parallel base layer

@hadipash filed RFC #2363 (2026-03-31) and Draft PR #2322, both implementing layer-wise PP for Wan2.2. After reviewing the #2322 source code directly, the relationship is as follows.

What #2322 provides and this RFC builds on:

• wan2_2_transformer.py: block-range dispatch via make_layers + PPMissingLayer + IntermediateTensors — this is the correct layer slicing mechanism; Stream Batch builds on it directly without duplication
• Wan22Pipeline inheriting PipelineParallelMixin alongside CFGParallelMixin — the right Mixin integration point, consistent with existing codebase conventions
• PP group initialization in group_coordinator.py

Where the two diverge (by design):

#2322's scheduler_step_maybe_with_pp_and_cfg sends updated latents from the last rank back to rank 0 (blocking on handles) before the next step can begin — a sequential per-step round-trip. Stream Batch eliminates this round-trip entirely: rank 0 immediately starts processing the next latent while the last rank emits a clean output frame. These are fundamentally different execution models; the sequential handoff cannot be reused for Stream Batch.

Additionally, make_layers partitions transformer blocks uniformly. compute_balanced_block_split (this RFC) adds VAE-overhead-aware uneven partitioning to avoid pipeline stalls on boundary ranks — this is purely additive and does not conflict with #2322's infrastructure.

Coordination plan: Since #2322 is still a draft, Stream Batch is developed in parallel, building on #2322's transformer layer slicing. One concrete alignment item before either PR merges: the PP group init should support combined PP+SP world size for the hybrid mode flagged in Q2.

---

5. Configuration Interface

Following the DiffusionCacheConfig pattern (see vllm_omni/diffusion/config.py):

diffusion:
  parallelism:
    strategy: pipeline        # "none" | "sequence" | "pipeline" | "auto"
    pipeline_stages: 4
    stream_batch_steps: 4
    vae_placement: inline     # "inline" | "shared" (requires #2089)
    block_partition: dynamic  # "uniform" | "dynamic"
  slo:
    target_fps: 30
    max_ttff_ms: 500
    adaptive_steps: true
  hardware:
    topology: auto            # "nvlink" | "pcie" | "auto"

Environment variable override (following the DIFFUSION_* naming convention established in #1568):

DIFFUSION_PARALLELISM_STRATEGY=pipeline
DIFFUSION_PIPELINE_STAGES=4
DIFFUSION_STREAM_BATCH_STEPS=4

---

6. Implementation Roadmap

Phase 1 — Foundation (already complete)

• #1368 (merged): step-wise runner interface
• #1625 (merged 2026-03-23): Scheduler/Executor boundary

Phase 2 — Core (this RFC, P1)

Base layer (provided by #2322): PipelineParallelMixin, transformer layer slicing (make_layers + PPMissingLayer + IntermediateTensors), PP group init. Stream Batch is developed in parallel and builds on this infrastructure without modifying it.

• StreamBatchScheduler: replaces #2322's sequential per-step latent round-trip with n latents simultaneously in-flight; runs above PipelineParallelMixin without modifying it
• compute_balanced_block_split + optional BlockScheduler (VAE-overhead-aware partition; additive to #2322's uniform make_layers split)
• StreamVAE (chunk-wise, 3D conv feature caching)
• p2p_comm.py shared P2P primitives in distributed/ (dedicated com_stream for compute/comm overlap)
• Opt A: Recv-side dist.irecv double-buffering (async_recv flag)
• Opt B: Eliminate redundant original_latents per-step transmission
• Reference integration: Wan2.1-1.3B (building on #2322's Wan2.2 integration)
• Target: 40+ FPS on 4×H100 (480p), ~16 FPS on 4×RTX 4090

Phase 3 — Integration (P1)

• DistributedKVCache stub (full impl deferred to #1987)
• #2089 VAE separate stage integration
• Multi-model: Wan2.2, HunyuanVideo-1.5
• PCIe topology path
• YAML config + env var system

Phase 4 — Stretch (P2)

• Full DistributedKVCache with rolling window for AR diffusion
• SemiCausalAttentionBackend (joint item with #2226)
• Benchmark suite: FPS / TTFF / GPU utilization vs SP baseline

---

7. Open Questions

Q1: Stream Batch interacts with CFG parallel (#851). If we run 2 CFG branches, does the effective batch double?

In StreamDiffusionV2, both CFG branches are treated as part of the effective batch B. This needs explicit design when integrating with vLLM-Omni's existing CFG Parallel implementation.

Q2: Is pipeline_stages == num_gpus always optimal? What about PP+SP hybrid?

For a single stream: more PP stages → higher FPS (more steps in-flight). For multi-stream: pipeline_stages=2, dp_replicas=2 may give better aggregate throughput.

However, @asukaqaq-s reports that for 14B models on H100, SP=4 is needed to reach 16fps — PP=4 alone is insufficient. This suggests the optimal real-time config for 14B models may require a PP+SP hybrid. The group initialization interface should be designed to support combined PP+SP from the start. This is a key open design question for Phase 2.

Q3: Which Phase 2 optimizations should be in scope?

• Opt A (dist.irecv double-buffering): self-contained, high value on PCIe. Include in Phase 2.
• Opt B (eliminate redundant original_latents): simple protocol change, ~15% per-step bandwidth reduction for n>1 steps. Include in Phase 2.
• Opt C (VAE broadcast → targeted P2P): depends on per-model cross-attention analysis. Phase 3.
• Opt D (timing data → #290 EDF scheduler): requires #290 to define an interface first. Phase 4.

Q4: How does this relate to RFC #290 (Diffusion Chunked Scheduling)?

The two are complementary. #290 addresses multi-request HoL blocking via EDF preemption (which request runs next). This RFC addresses single-stream GPU utilization via PP + Stream Batch (how to keep all PP stages busy). In production both coexist: StreamBatchScheduler can expose its per-frame latency estimates to #290's EDF slack calculation.

Q5: What is the distinction between PP stages in this RFC and Stages in the vLLM-Omni stage graph?

These are two orthogonal concepts:

• Omni Stage (from the stage graph): a model-level unit in the multimodal pipeline — e.g., LLM Thinker, DiT Generator, VAE, TTS Talker. Each Omni Stage is an independently scheduled engine.
• PP stage (this RFC): a rank within the DiT Generator Omni Stage, holding a contiguous slice of DiT transformer blocks.

The StreamBatchScheduler runs once per DiT Omni Stage instance (on rank 0); other PP ranks follow its micro-step schedule. The two levels are orthogonal: each Omni Stage can have its own PP degree independently.

Q6: Is the ~30% VAE overhead figure reliable?

@asukaqaq-s notes VAE does not account for a large proportion in his Torch 2.8.0 tests — the ~30% figure may reflect a slow Conv3d path present in newer PyTorch versions. The compute_balanced_block_split overhead parameters (vae_encode_overhead=0.15) are empirical defaults that should be profiled per deployment. The two-phase BlockScheduler (runtime all_gather timing) exists precisely to handle cases where the static estimate is wrong.

---

8. References

• StreamDiffusionV2 (arXiv:2511.07399, MLSys 2026) — primary design reference; code: chenfengxu714/StreamDiffusionV2; built on Wan2.1 + CausVid distillation
• xDiT (arXiv:2411.01738, NeurIPS 2025) — multi-GPU DiT inference; Performance Reports: github.com/xdit-project/xDiT/docs/performance/
• PipeFusion (arXiv:2405.14430, NeurIPS 2025) — patch-level PP for offline DiT (distinct from this RFC)
• #874 — Step-wise Batching; core interfaces in #1368 + #1625 (both merged)
• #290 — Diffusion Chunked Scheduling (complementary, not conflicting)
• #1987 — World Model Support (consumer of this RFC)
• #2089 — VAE as Separate Stage (long-term VAE solution)
• #647 — PipeFusion Implementation (shares P2P comm primitives)
• #268 — Async Stages (orthogonal: inter-stage orchestration)
• #2363 — Add support for Pipeline Parallel (RFC by @hadipash; same base mechanism, offline memory use case)
• #2322 — [Draft PR] Add support for Pipeline Parallel and integrate into Wan 2.2 (by @hadipash; provides PipelineParallelMixin + transformer layer slicing; Stream Batch scheduler builds on this)
• #2226 §3 — Q2 Roadmap entry

---

9. CC

@hsliuustc0106 @wtomin @asukaqaq-s @SamitHuang @ZJY0516 @TKONIY @Gaohan123 @fhfuih @david6666666

---

Version: v0.2.0 | Date: 2026-03-27 | Priority: P1 (Q2 2026, see #2226 §3) | Parent: #2226 | Related: #874 #1987 #2089 #647 #268 #290 #2363 #2322 | Distinct from: -- #647 (PipeFusion — patch-level PP for offline), -- #268 (Async Stages — inter-model orchestration), -- #290 (Chunked Scheduling — multi-request EDF)