[RFC]: HunyuanImage 3.0 deployment

Motivation.

#1085 #759 HunyuanImage 3.0 is currently the state-of-the-art (SOTA) open-source model in the image generation and editing domain. It utilizes a massive 80B MoE backend via a Transfusion architecture. In vllm-omni, while the base DiT component has been merged (with AR pending) and Tensor Parallelism (TP) is supported, we face severe bottlenecks when scaling to production-grade disaggregated deployment:

(1) Throughput and Efficiency under Multi-Request Batching: The heavy computational burden of running an 80B MoE iteration loop (50 steps) requires significant optimization. The current standard batch duplication (torch.cat([latents] * cfg_factor)) for Classifier-Free Guidance (CFG) pads all sequences to the longest one along the batch dimension and uses an explicit (bsz, 1, seqlen, seqlen) attention mask with F.scaled_dot_product_attention. While for CFG itself the conditional and unconditional branches are equal-length (no intra-CFG padding waste), the real inefficiency emerges when serving concurrent requests at different resolutions: short-sequence requests are padded to match the longest request in the batch, wasting O(seqlen²) compute in attention and activating unnecessary MoE experts on padding tokens. Additionally, the materialized 4D attention mask consumes quadratic memory per batch element. (2) The Pipeline Parallelism (PP) Dilemma: Storing and updating a full 80B MoE pushes VRAM boundaries. While the model authors desire a TP + PP + UP combination, applying PP to a DiT loop is highly problematic. It introduces extreme cyclic communication overhead because the latent tensor must be transferred from the last PP rank back to the first PP rank repetitively for 50 inference steps. A robust memory scaling alternative (e.g., Weight Offloading) must be evaluated and implemented. (3) High-Resolution Scaling: The current implementation lacks support for Ulysses Parallelism (UP) and Ring Attention, which are strictly required for handling the massive token sequence lengths generated in high-resolution/4K image editing tasks.

For HunyuanImage 3.0, our goal is to solidify the DiT inference engine, finalize the cross-stage KV transfer, and introduce advanced parallelism (Packed-Attn, UP, Offloading) to maximize hardware utilization and achieve the speed needed for production viability.

Proposed Change

P0: Core Architecture & Foundation

• Packed Attention for Multi-Request Batching (DiT Stage) [Optimization] #1975
  • Replace the naive latent_model_input = torch.cat([latents] * cfg_factor) batch duplication and the explicit (bsz, 1, seqlen, seqlen) attention mask.
  • Implement a contiguous sequence packing approach (packed-attn / cu_seqlens) using flash_attn_varlen_func, allowing requests of different resolutions to be packed into a single forward pass without padding waste. This is the primary optimization target.
  • Clarification on CFG: Unlike Bagel where three CFG branches have heterogeneous KV lengths (gen=VAE+ViT+prompt, cfg_text=VAE+ViT+neg_prompt, cfg_img=prompt only), HunyuanImage's cond/uncond branches share identical sequence lengths within a single request. Therefore the packed-attn gain here is not about eliminating intra-CFG padding, but about enabling efficient cross-request batching for mixed-resolution workloads and eliminating the quadratic memory overhead of the explicit attention mask tensor.
  • Attention mask challenge: HunyuanImage uses Generalized Causal Attention (text tokens = causal, image tokens = full bidirectional). Expressing this mixed pattern in varlen flash attention requires either splitting into separate causal/full segments or constructing a block-sparse mask, adding implementation complexity.
• KV Cache Transfer Integration for DiT [Feature]
  • Finalize the integration of OmniKVTransferManager within the DiT pipeline to seamlessly receive and deserialize the massive Text Prefix context from the AR stage, ensuring zero re-computation of prompt/image conditions during the denoising loop.

P1: Performance & Production Readiness

• Pipeline Parallelism (PP) vs. Layerwise Weight Offloading [Architecture]
  • Conduct a definitive overhead analysis to resolve the memory-wall issue.
  • Context: We think PP communication overhead is excessively high due to the Stage N → Stage 0 cyclic transfer per timestep.
  • Action: Prioritize and implement a highly optimized Layerwise Weight Offloading pipeline (using asynchronous prefetching streams from CPU RAM) as the primary alternative to PP for memory-constrained environments.
• CFG-Parallel Implementation [Feature] #1751
  • Implement parallel dispatch logic specifically allowing DiT to calculate the Cond branch and Uncond branch synchronously across independently split GPUs, followed by a fast cross-node gathering reduction.
• TEA Cahe Implementation [Feature] #1927
• Tiling encode [Feature] #1953
• EP [Feature] #1323

P2: Advanced Scalability

• Ulysses Sequence Parallelism (UP) Implementation [Feature]
  • Implement Ulysses SP for the HunyuanImage 3.0 backend. By partitioning the spatial image tokens across GPUs, we can gracefully support massive resolution extensions without OOM.
• Ring-Attention Integration [Feature]
  • Implement Ring-Attention as a complementary feature to UP, further supporting extreme context windows required by complex text-to-image or image-to-image editing tasks.
• EPLB [Feature]

Feedback Period.

No response

CC List.

@hsliuustc0106 @princepride @ZJY0516 @natureofnature @lishunyang12 @yuanheng-zhao @ElleElleWu

Model structure.

Core difference between HunyuanImage 3.0 vs. Bagel

To contextualize the optimization focus, the fundamental structural differences between this model and our recent Bagel implementation must be highlighted:

1. Native Transfusion (Single Prefix KV) vs. Frankenstein Contexts (Decoupled Trees)
   • Bagel acts as a decoupled router, utilizing 5 serial forward and deepcopy operations to compute varying contexts (Text-only, Text+Img, etc.), and managing them concurrently during the DiT phase.
   • Hunyuan is a native multi-modal model (Transfusion architecture). The Text, VAE, and ViT tokens are packed dynamically into a singular, immensely long context array using Generalized Causal Attention (where image patches can fully attend to each other simultaneously). Thus, Omni only needs to transfer and manage one massive Prefix KV, rather than multiple isolated feature trees.
2. CFG Batched Splitting (Hunyuan) vs. cu_seqlens Muxing (Bagel)
   • Because of its complex tree, Bagel computes CFG inside DiT by leveraging cu_seqlens and packed-attn natively, cleverly multiplexing shared context inside a monolithic KV structure to avoid batch duplication.
   • Hunyuan currently uses crude batch duplication (torch.cat([latents] * cfg_factor)) prior to the MoE. This doubles runtime VRAM activations and introduces vast padding waste. Bringing our Bagel-style packed-attn methodology into Hunyuan is our highest P0 priority for eliminating this specific bottleneck.
3. MoE Dominance in DiT (13B Active vs. 7B Dense)
   • Bagel runs diffusion on a relatively dense standard backend, meaning compute resources are fairly uniform per sequence length.
   • Hunyuan's 80B DiT loop heavily relies on MoE (activating ~13B per step). Without sophisticated Expert Parallelism (EP) intersecting with TP/UP inside the 50-loop timestep cycle, specific nodes will easily face Out-Of-Memory (OOM) deadlocks or massive compute bubbles.

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.