RFC: World Model Support

1. Overview

World Models predict how the world evolves in response to actions — enabling real-time interactive loops: continuous input (observations, actions, control signals) → continuous output (future video, predicted actions).

Two application domains:

• Robotics Control: DreamZero, Motus, LingBot-VA (on video DiT), Pi0, OpenVLA (on VLM) — predict robot actions at up to 7Hz.
• Interactive Video / Simulation: Genie 3, Matrix Game, HunYuan World, LingBot-World, Cosmos WFM — controllable video for games, digital twins, simulators.

Scenarios

Scenario	Input	Output	Models
Robotics	Multi-view images + text + robot state	Actions (± video)	DreamZero, Motus, Pi0, OpenVLA
Interactive Video / Game	Image + text + control (camera, joystick, keyboard)	Video stream	Matrix Game, HunYuan World, Genie 3
Simulation	Scene state + action sequence	Video Stream	Cosmos WFM, LingBot-World

---

2. What's New vs Current vLLM-Omni

Input/Output — Action modality. Current models produce text/audio/image/video. World Models additionally consume and produce action sequences (robot joint angles, game controls), multi-view images, and embodiment state.

API — Realtime streaming with state management. Current streaming is single-turn unidirectional (TTS: text→audio). World Models need a multi-turn loop: client pushes observations, server returns predictions, repeat. Neither vllm-omni's TTS WebSocket nor vLLM's Realtime API support multi-turn context accumulation.

Engine — Autoregressive diffusion. Current diffusion denoises the full sequence in one pass. Many world models introduce autoregressive chunk diffusion: each forward produces one video chunk while attending to all past chunks. The attention is blockwise causal.

Note: Stateless models (Motus) and VLA models (Pi0, OpenVLA) do NOT require the state or engine changes — they work with existing infrastructure plus action I/O support.

---

Roadmap

Stage 1: Robotics

Goal: the first world model support, with architecture for robotics and multiturn realtime service.

• Support DreamZero

  • P0: #2063
    • P2: (side) #2160 #2078
    • P2: (side) #2423
  • P0: Multiturn stateful session management P0: Support DreamZero-Droid P0: OpenPI style API server (realtime api) #2162 #3673

• P1: Lingbot-VA #2885

• P1: Performance Optimizations (DreamZero first, applied for all WAMs) #4127

• General robotics support for datasets/platform #3554

• Embodied Models Support

  • P2: Supporting lingbot-VA and other WAMs
  • P2: Supporting VLAs, e.g., pi,
    • #1948
    • #3471
    • #3553

• RL support #3747

  • Integration with RLinf.

Future (Discussion needed)

• Interactive video generation models, APIs, optimizations and RL.
  • #2126
• Page-attention and KV cache management for Autoregressive Diffusion.

Feedback Period.

TBD

CC List.

@yinpeiqi @ywang96 @Gaohan123 @hsliuustc0106 @asukaqaq-s @IlyasFardaouix @matchyc

Related Issues

• #1045
• #1948
• #2073
• #874
• #3454
• #4126