[RFC]: LingBot-World-Fast — Interactive video world model port

Motivation

Port LingBot-World-Fast (Robbyant/lingbot-world, Apache 2.0) to vllm-omni as a new diffusion pipeline under vllm_omni/diffusion/models/lingbot_world_fast/. LingBot-World-Fast (LWF-Fast) is the accelerated, streaming variant of LingBot-World (arXiv 2601.20540), a single-image → camera-controlled interactive video world model built on the Wan 2.2 A14B DiT backbone. This RFC plans the first supported model for the interactive-video track of RFC #1987, and sits alongside the robotics-control track represented by PR #2885 (LingBot-VA). The model is Apache-2.0 end-to-end, fits a single 80 GB device in BF16 at 480×832, and reuses the Wan 2.2 DiT primitives, Wan 2.1 VAE, and UMT5 text encoder already in-tree.

Proposed Change

1. Overview

LWF-Fast takes a single image, a text prompt, and a camera trajectory (per-frame intrinsics + poses) and produces a video rendered from the camera's point of view, streamed chunk-by-chunk in real time. The model is a causal autoregressive DiT (distilled to 4 inference steps per chunk), chunk-decoded three latent frames at a time, with two per-layer KV caches (self-attention andcross-attention) persisted across chunks in a per-session state container. Camera conditioning is injected per block as AdaLN-style scale+shift from a per-pixel 6-dimensional Plücker ray representation (get_plucker_embeddings(intrinsics, poses)).

LWF-Fast supports exactly one control modality: camera poses. It is not action-conditioned; the sibling LingBot-World-Base (Act) checkpoint that accepts WASD keyboard inputs is a separate, not-yet Apache-unblocked release and is explicitly out of scope here (see §2.2). This is confirmed by the upstream Model Download table at github.com/Robbyant/lingbot-world (Base (Cam): Camera Poses; Base (Act): Actions; Fast: Camera Poses) and by the 4-step Fast pipeline's control_type='cam' code path in wan/image2video_fast.py.

Serving shape. LWF-Fast is exposed through a realtime WebSocket endpoint at /v1/realtime/world/camera, aligned with the interactive-streaming API pattern already established by PR #2162 (DreamZero's /v1/realtime/robot/openpi) and by vllm-omni's TTS realtime endpoint. The URL names the protocol family — "realtime camera-trajectory → video" — not the model (following PR #2162's precedent, where /robot/openpi names the wire shape, not DreamZero specifically; the loaded model is determined by the vllm serve <model> CLI at startup). A client opens a session, receives a server config handshake, then sends one camera-pose request per chunk and receives one encoded video chunk back; the session's KV cache and frame buffer persist across requests for the lifetime of the connection. A request/response /v1/videos convenience wrapper (bundling a full precomputed trajectory into one session under the hood) is a natural follow-up for offline / simulation use; not in v1.

Context: LWF-Fast complements PR #2885 (LingBot-VA) by establishing the other endpoint of RFC #1987's "world model" scenario table. Both share the Wan 2.2 backbone, Wan 2.1 VAE, UMT5 text encoder, and a session-scoped KV-cache streaming pattern. LWF-Fast adopts the realtime-WebSocket + session-state-container serving shape that PR #2162 introduced; the per-session state container (holding KV cache, frame buffer, and RoPE current_start offset) is the vllm-omni precedent we align with. No engine-level KV-cache abstraction is introduced here; a unified PageAttention-style KV manager for autoregressive diffusion is explicitly RFC #1987's downstream plan, not this PR's concern.

2. Scope & Objectives

2.1 In scope for this PR

• New diffusion model directory at vllm_omni/diffusion/models/lingbot_world_fast/ with the Fast-variant transformer, pipeline, weight loader, per-session state container, and single-GPU stage config.
• Realtime WebSocket endpoint at /v1/realtime/world/camera, mirroring the shape PR #2162 introduced for DreamZero (msgpack binary framing, handshake with a model-specific ServerConfig, per-chunk infer and reset endpoints within one session). Serving handler under vllm_omni/entrypoints/openai/realtime/world/.
• Camera-pose control carried in the per-chunk request payload as cameras: {intrinsics: [N,4], poses: [N,4,4]} (msgpack dict). OpenCV convention, ViPE / Nerfstudio / COLMAP compatible. Text prompt carried as prompt: str in the same payload.
• Session-scoped state container (LingBotWorldFastState) owning: the two KV caches (self + cross, per-layer) with their _pos / _neg CFG-parallel variants, the frame buffer, the RoPE current_start offset, and the reset logic triggered by session-id change or prompt change. Modelled on DreamZeroState from PR #2162.
• 4-step distilled inference, schedule applied at pipeline setup against the in-tree FlowUniPCMultistepScheduler (timestep indices [0, 179, 358, 679] out of 1000; no new scheduler class).
• Chunked causal decoding with an explicit cache-update forward at t≈0 on clean latents after each chunk; chunk boundaries are client-triggered (one infer message per chunk). Chunk size 3 latent frames (~12 video frames at 4× VAE temporal stride).
• CFG-parallel support via in-tree CFGParallelMixin (two caches: _pos / _neg within the state container).
• BF16 correctness, SDPA attention baseline, FA2/FA3 auto-used when available via the in-tree attention-backend selection.
• TP-parallel linear wiring preserved from the in-tree Wan 2.2 primitives we compose against. Not a strip-and-fork.
• Example usage under examples/online_serving/video_generation/: one WebSocket Python client showing handshake + per-chunk infer loop with intrinsics / poses trajectory playback and base64-MP4-chunk reassembly.

2.2 Out of scope for this PR (explicit deferrals)

• LingBot-World-Base (Cam) — bidirectional diffusion, different denoising schedule, distinct checkpoint. Same model family, follow-up port.
• LingBot-World-Base (Act) — action-conditioned (WASD multi-hot concatenated with ray directions), distinct checkpoint, reference inference is generate.py with --action_path / --action_string /--allow_act2cam. Follow-up port; would extend the protocol with a sibling field (e.g. actions: {keys: […]}) or a discriminated control union.
• Engine-level / unified KV-cache service for AR diffusion. Session-scoped state container in v1, modelled on PR #2162's DreamZeroState and adjacent to in-tree glm_image_transformer's module-resident cache and PR #2885's attn_caches dict. A unified PageAttention-style KV manager for AR diffusion is RFC #1987's stated downstream plan; this PR adopts the current workaround shape, not the future abstraction.
• Sequence-parallel (Ulysses SP=8). Reference upstream config is SP=8; we land single-GPU first. SP follow-up depends on how RFC #2322 (hadipash's wan2.2 PP draft) and the communication-layer refactor (RFC #1546) settle.
• FA3 explicit wiring, INT8 attention, INT8 DiT, LightVAE-style VAE distillation, compiled VAE. All are separate optimisation PRs orthogonal to this port. Paper's 40 FPS steady-state on 8× H100 is attained with the product of these; we state the gap but don't chase it in v1 (see §3.8).
• Request/response /v1/videos convenience wrapper. Bundling a precomputed trajectory into one session server-side is a natural follow-up for offline / simulation callers. Not v1; realtime WebSocket is the primary (and only) serving shape we ship now, matching PR #2162's precedent.
• Dormant rolling-window eviction code. Released checkpoint has local_attn_size=-1, sink_size=0; eviction path is gated on local_attn_size != -1 and off by default in v1.

2.3 Correctness criteria

• A WebSocket session that plays the authors' examples/04 inputs (image + 533-pose trajectory + Great-Wall prompt) as 27 sequential infer requests produces a reassembled 81-frame, 480×832 video visually consistent with the reference generate_fast.py offline output at the same seed, within numerical tolerance expected from stochastic 4-step flow-match sampling (no strict bit-parity claim).
• A shorter variant of the same session (9 infer requests, 25 frames) produces a reference-comparable video without temporal collapse or scene-identity loss.
• The realtime endpoint's handshake returns a well-formed CameraServerConfig (msgpack) on connect; infer requests with malformed cameras payloads return a typed {type: "error"} frame rather than closing the connection; a reset request clears all session state (caches, frame buffer, RoPE offset).
• Session state persists correctly across chunks — the same trajectory played as one 27-chunk session produces the same output as the reference's single offline invocation on the full trajectory (within BF16 numerical tolerance).
• Peak single-GPU VRAM at 480×832 over a full 81-frame session stays below 70 GiB; at 25 frames, below 50 GiB.
• Cache-update pass produces the same post-chunk KV state as would be obtained by a full forward on clean x0 (invariant, not a performance target — see §4.1).

3. Design

3.1 Pipeline shape

client ──────────────────► server: WS upgrade  /v1/realtime/world/camera
server ──msgpack(CameraServerConfig)──► client
                                         (patch_size, chunk_frames,
                                          fps, image_resolution, ...)

   ┌─── one session, N chunks ─────────────────────────────────────┐
   │                                                                │
   │ client ──msgpack({endpoint:"infer",                            │
   │                   session_id, image?, prompt?,                 │
   │                   cameras:{intrinsics, poses}})──► server      │
   │                                                                │
   │    ┌──── ServingRealtimeWorldCamera.infer() ──────┐           │
   │    │                                                │          │
   │    │  [first chunk of session only:]                │          │
   │    │   T5 encode(prompt)      ─► cross KV cache     │          │
   │    │   VAE encode(image)      ─► y (first latent)   │          │
   │    │                                                │          │
   │    │  [every chunk:]                                │          │
   │    │   get_plucker(K_slice,                         │          │
   │    │               poses_slice)                     │          │
   │    │                 ─► Plücker emb [B,6,f,H/2,W/2] │          │
   │    │   current_start = state.global_end_index       │          │
   │    │                                                │          │
   │    │   for t in [0, 179, 358]:  (3 noisy steps)     │          │
   │    │     eps = WanModelFast(x, t, plucker,          │          │
   │    │                        self_kv_cache,          │          │
   │    │                        cross_kv_cache,         │          │
   │    │                        current_start, ...)     │          │
   │    │     x0 = flow_pred_to_x0(eps, x, t)            │          │
   │    │     x  = add_noise(x0, randn, next_t)          │          │
   │    │   # last step: keep x0 (no re-noise)           │          │
   │    │                                                │          │
   │    │   # cache-update pass, t ≈ 0, clean x0         │          │
   │    │   WanModelFast(x0, t≈0, plucker,               │          │
   │    │                self_kv_cache, ..., update=True)│          │
   │    │                                                │          │
   │    │   state.append_chunk(x0)                       │          │
   │    │   video = VAE.decode(x0)                       │          │
   │    └────────────────────────────────────────────────┘          │
   │                                                                │
   │ server ──msgpack({type:"frame", chunk_id, video:<mp4 bytes>,   │
   │                    stage_durations:{...}})──► client           │
   │                                                                │
   │ (repeat for next chunk; session-scoped                         │
   │  LingBotWorldFastState persists self_kv_cache,                 │
   │  cross_kv_cache, frame buffer, current_start,                  │
   │  CFG _pos/_neg variants across chunks)                         │
   │                                                                │
   │ client ──msgpack({endpoint:"reset"})──► server                 │
   │   state.reset() clears caches + frame buffer                   │
   │                                                                │
   └────────────────────────────────────────────────────────────────┘

3.2 Model architecture

From the released lingbot_world_fast/config.json + wan/modules/model_fast.py:

dim=5120, num_heads=40, num_layers=40, ffn_dim=13824
patch_size=(1,2,2), in_dim=36, out_dim=16
control_type='cam'      → Plücker camera embeddings (6-d per voxel)
local_attn_size=-1
sink_size=0             → global attention, no rolling eviction (v1)

in_dim=36 = 16 (noise latent) + 16 (image conditioning) + 4 (temporal mask). Input is concatenated on the channel axis.

Per-block additions vs. stock Wan 2.2:

• PluckerCamInjector (new module) — AdaLN-style scale+shift injection of the Plücker embedding at every transformer block. Four Linear(dim, dim) layers: cam_injector_layer1/2 (2-layer SiLU MLP residual), cam_scale_layer, cam_shift_layer.
• CausalWanSelfAttention (new module) — self-attention reading from and writing to a module-resident KV cache, with an externally-supplied current_start offset fed to RoPE so chunk-to-chunk positions remain continuous.

Top-level additions:

• patch_embedding_wancamctrl: Linear(6·64·∏patch_size, dim) — per-patch Plücker embedding projector. The constant 64 is the tiled-voxel count (trace during port; see §5 open questions).
• c2ws_hidden_states_layer1/2 — 2-layer MLP residual on the Plücker embedding prior to per-block injection.

Unchanged composition points (imported from vllm_omni/diffusion/models/wan2_2/wan2_2_transformer.py):

• WanRotaryPosEmbed — 3D RoPE.
• WanTimeTextImageEmbedding — timestep / text / image-condition embedding bundle.
• WanFeedForward — per-block FFN.
• WanTransformerBlock shape, TP linear wiring (ColumnParallelLinear, RowParallelLinear, QKVParallelLinear), sp_plan conventions.

3.3 Inference loop (detail)

Already shown in §3.1 as a flow diagram. Noteworthy details:

• Chunk boundary is client-triggered. One infer WebSocket message per chunk. Same driver model as PR #2162. The server does not autonomously emit frames; it produces one chunk per received infer and waits.
• 4 inference steps per chunk. The schedule is applied at pipeline setup time as indices into FlowUniPCMultistepScheduler(num_train_timesteps=1000).timesteps; no new scheduler class.
• Cache-update pass is a deliberate separate forward at t≈0 on the final clean latent. Ensures the KV cache for the next chunk sees clean tokens, not intermediate noisy ones.
• Text encoder runs once per session. On the first infer of a session, the prompt is T5-encoded and the cross-attention cache is populated; crossattn_cache.is_init gates the reuse from chunk 2 onward. A reset clears it; a prompt change within a session triggers an implicit reset (matching DreamZeroState.should_reset() behaviour).
• CFG-parallel: positive and negative branches each maintain their own self_kv_cache and cross_kv_cache, per CFGParallelMixin convention. Cache-update pass runs on both.
• Session state persists across chunks within one WebSocket connection. Closing the connection drops the state; opening a new session ID on a live connection resets it. Both paths are the same state.reset() call.

3.4 API surface — /v1/realtime/world/camera WebSocket

The serving shape is a realtime WebSocket with msgpack binary framing, directly modelled on PR #2162's /v1/realtime/robot/openpi endpoint (see vllm_omni/entrypoints/openai/realtime/robot/openpi_connection.py and openpi_serving.py). We adopt that precedent verbatim where possible and diverge only in payload shape (camera poses instead of robot observations; video chunks instead of action tensors).

Upgrade and handshake.

GET /v1/realtime/world/camera     (HTTP → WebSocket upgrade)

On accept, the server immediately sends a msgpack-serialised CameraServerConfig frame advertising model capabilities:

CameraServerConfig = {
  "patch_size":        [1, 2, 2],
  "chunk_latent_frames": 3,
  "chunk_video_frames":  12,          # 4× VAE temporal stride
  "vae_temporal_stride": 4,
  "image_resolution":   [480, 832],   # H, W
  "frame_dtype":        "uint8",
  "fps":                16,
  "needs_session_id":   true,
  "supported_modalities": ["camera_pose"],
  "coordinate_convention": "opencv_c2w"
}

Shape and framing follow PolicyServerConfig in PR #2162 (openpi_serving.py:34). Field set is LWF-specific; mechanism is identical.

Per-chunk infer request (client → server).

{
  "endpoint":   "infer",              # or "reset"
  "session_id": "<uuid string>",
  # required on the first infer of a session only:
  "image":     <bytes, png/jpg> | None,
  "prompt":    "<text prompt>"        | None,
  # required on every infer:
  "cameras": {
    "intrinsics": [[fx, fy, cx, cy], ...],   # [N, 4]  float32
    "poses":      [[[...], ...]],            # [N, 4, 4] OpenCV c2w
  },
}

• N per chunk covers chunk_video_frames output frames; intrinsics and poses may be over- or under-specified and are interpolated to chunk_latent_frames latent frames by the pipeline (get_Ks_transformed + linear pose interpolation), matching the reference implementation.
• image + prompt are session-scoped: accepted on the first infer of a new session_id, ignored on subsequent chunks (or trigger an implicit reset if they change, matching DreamZeroState.should_reset() semantics).
• Validation. Shape / dtype checked; malformed payloads return an error frame ({type: "error", message: ...}, msgpack) without closing the connection. cameras is required on every infer.

Per-chunk response (server → client).

{
  "type":     "frame",
  "chunk_id":  0,                      # monotonic per session
  "video":    <bytes, mp4 fragment>,
  "stage_durations": { ... },          # per-stage timings
}

Video payload is an MP4 fragment holding chunk_video_frames frames at fps. Client reassembles by concatenation (fragmented-MP4 conventions). Rationale for MP4 over raw RGB: keeps the wire small (~100–500 KB per chunk at 480×832 vs. ~20 MB raw) and requires no codec work on either side beyond stdlib av / ffmpeg.

reset request (client → server).

{ "endpoint": "reset", "session_id": "<uuid>" }

Server calls state.reset(), drops all KV caches and the frame buffer for this session, and replies with a confirmation: {type: "reset_ok", session_id: "..."}. Next infer on the same session_id re-runs the first-chunk initialisation (T5 encode, VAE encode of initial image).

Session semantics. One session per session_id. A single WebSocket connection may carry one session (the common case) or multiple sequential sessions separated by reset. Server-side state lives on the pipeline (one LingBotWorldFastState instance per process, mirroring PR #2162's DreamZeroState); session_id is a change-detector — a fresh value triggers state.reset() on the next infer, matching DreamZero's serving-layer behaviour. The v1 server is single-tenant: one active session at a time per process. Connection loss drops the shared state via a reset() call on the disconnect path. (Multi-tenant operation — concurrent sessions on one server, isolated per session_id — is out of scope for v1; see §3.5 decision 8. Aligned with the deferred session-management RFC referenced on #1987 by @tysoncung / @Sy0307, 2026-03-23.)

Error frames. Non-fatal errors (malformed payload, missing required field on first-chunk, unsupported cameras shape) are returned as {type: "error", code: <str>, message: <str>} and do not close the connection. Fatal errors (unrecoverable state, protocol violation) close with a 1011 code and a text reason.

Why not /v1/videos multipart. RFC #3072 v1 proposed a /v1/videos multipart form extension. Reviewer feedback (TKONIY on #3072, 2026-04-23) directed the design to mirror the realtime streaming precedent set by PR #2162 and vllm-omni's TTS realtime endpoint, because /v1/videos is request/response-only and cannot carry a multi-chunk streaming session. We adopt the WebSocket shape as the single v1 API; an offline /v1/videos convenience wrapper that plays a full precomputed trajectory as one session server-side is a natural follow-up, not blocking v1.

To the best of our knowledge, no published HTTP schema exists for full-pose camera-controlled video streaming. We ground the camera field shape in established research-tooling conventions (Nerfstudio / ViPE / COLMAP) and the framing / session conventions in PR #2162's precedent. No vllm-omni-specific convention is invented beyond the concrete field names.

3.5 Key decisions, with rejection rationale

1. Composition, not fork, of the Wan 2.2 transformer. The in-tree wan2_2/wan2_2_transformer.py already has TP-parallel linear wiring, sp_plan integration, attention-backend wiring, and AutoWeightsLoader. Importing those primitives and adding PluckerCamInjector and CausalWanSelfAttention as new composable modules preserves all of it. Rejected alternative: strip the TP/SP/attention-backend code and fork the transformer. Rejected because it discards machinery we would then need to re-add piecemeal to scale beyond 1× GPU.
2. Session-scoped state container holding the KV caches, adopted directly from PR #2162's DreamZeroState. One LingBotWorldFastState instance per session, owning: per-layer self-KV dicts {k, v, global_end_index, local_end_index}, per-layer cross-KV dicts with an is_init flag, their _pos / _neg CFG-parallel variants, a frame buffer for output assembly, and the RoPE current_start offset. The pipeline threads the state through each forward pass; the state persists for the lifetime of the session. Rejected alternative: a new engine-level KV-cache service with PageAttention-style paging. Rejected because RFC #1987 explicitly flags a unified AR-diffusion KV manager as the downstream plan (TKONIY, #3072 review),
3. Dormant rolling-window eviction. Released checkpoint has local_attn_size=-1, sink_size=0. Eviction code is gated on local_attn_size != -1 and off by default. Rejected alternative: delete the code path entirely. Rejected because it preserves the option to re-enable when a future checkpoint ships with local_attn_size ≠ -1.
4. Single in-tree scheduler, 4-step indices applied at setup (FlowUniPCMultistepScheduler). Rejected alternative: introduce a new LingBotFastScheduler class. Rejected because the 4-step schedule is a configuration of an existing scheduler, not a new algorithm.
5. Camera-pose-only control in v1. Rejected alternative: include a control_type='act' path now, with the WASD channel broadcast-concatenated to ray directions (the code delta is small). Rejected because the 'act' branch of image2video_fast.py is not backed by a publicly-released Fast checkpoint; the released weights are the cam-only model (confirmed by the upstream README's Model Download table). Base (Act) is a separate follow-up port against the Base (non-fast) model family.
6. msgpack over the WebSocket, not JSON. Rejected alternative: JSON framing. Rejected for wire consistency with PR #2162 (which uses msgpack throughout via the openpi_client.msgpack_numpy codec) and for efficiency on numeric payloads (pose / intrinsics arrays). We pick up msgpack's numpy extension for the camera arrays so clients can send float32 tensors natively without a JSON round-trip.
7. Realtime WebSocket as the v1 API shape. Rejected alternative: ship request/response /v1/videos first and add streaming later. Rejected on reviewer feedback (TKONIY, #3072, 2026-04-23) that /v1/videos does not support multiturn / streaming and that interactive world models should share the realtime API pattern with DreamZero (PR #2162), the TTS realtime endpoint, and upstream vLLM's Realtime API.
8. Single-tenant v1 server — one active session at a time per process, mirroring PR #2162. The LingBotWorldFastState instance lives on the pipeline as a flat object with scalar attributes; session_id is a change-detector that triggers state.reset() on mismatch (matching openpi_serving.py:120-125), not a lookup key. Cleanup on WebSocket disconnect is a single state.reset() call from the connection layer. Rejected alternative: Multi-tenancy is part of the downstream plan

3.6 Files and layout

vllm_omni/diffusion/models/lingbot_world_fast/
├── __init__.py
├── modeling_lingbot_world_fast.py   # transformer + blocks (composes wan2_2 primitives)
├── pipeline_lingbot_world_fast.py   # chunked inference, CFG-parallel, calls into state
├── state_lingbot_world_fast.py      # LingBotWorldFastState (session-scoped state container,
│                                    # modelled on DreamZeroState; see §3.5 decision 2)
└── plucker.py                       # get_plucker_embeddings port from wan/utils/cam_utils.py

vllm_omni/diffusion/registry.py
    # add "LingBotWorldFastPipeline" entry

vllm_omni/model_executor/stage_configs/lingbot_world_fast.yaml
    # single-GPU stage config (pre-install; stage configs are package data)

vllm_omni/entrypoints/openai/realtime/world/
├── __init__.py
├── camera_connection.py             # WorldCameraRealtimeConnection — WebSocket lifecycle,
│                                    # msgpack framing; modelled on openpi_connection.py
└── camera_serving.py                # ServingRealtimeWorldCamera — per-chunk infer,
                                     # session tracking, request building;
                                     # modelled on openpi_serving.py

vllm_omni/entrypoints/openai/api_server.py
    # add @router.websocket("/v1/realtime/world/camera") registration,
    # alongside the existing openpi_robot and TTS realtime routes

examples/online_serving/video_generation/
├── lingbot_world_fast_ws_client.py  # WebSocket client: handshake + per-chunk
│                                    # infer loop; reassembles fragmented MP4
└── lingbot_world_fast_trajectory.py # helper that loads the reference
                                     # examples/04 trajectory as msgpack-ready dicts

No separate scheduler file — indices-into-in-tree-scheduler are handled inside the pipeline (§3.5 decision 4). No /v1/videos server-side changes in v1 — the realtime endpoint is the only v1 serving surface.

3.7 Dependencies

• Upstream repositories. github.com/Robbyant/lingbot-world is the reference implementation. We vendor translated code, not dependency imports.
• HuggingFace checkpoints. robbyant/lingbot-world-fast (DiT shards, ~70 GiB), robbyant/lingbot-world-base-cam (T5 + VAE + tokenizer subset, ~10 GiB). Fast download ~80 GiB.
• In-tree dependencies. vllm_omni.diffusion.models.wan2_2.* (primitives — imported), vllm_omni.diffusion.distributed.cfg_parallel.CFGParallelMixin, vllm_omni.diffusion.distributed.autoencoders.autoencoder_kl_wan.DistributedAutoencoderKLWan, vllm_omni.diffusion.models.schedulers.scheduling_flow_unipc_multistep.FlowUniPCMultistepScheduler, vllm_omni.diffusion.attention.* (FA2 / FA3 / SDPA selection).
• No new third-party packages. The reference implementation's einops, numpy, torch dependencies are already in-tree.

3.8 Performance targets

Targets below are for the reference operating point: single A100 80 GB PCIe, BF16, SDPA or FA2 attention (both measured at parity; see below), 480×832 output resolution, 4 denoising steps, FlowUniPCMultistepScheduler with the [0, 179, 358, 679] timestep schedule, chunk size of 3 latent frames (≈ 12 video frames at the 4× VAE temporal stride). Each target is followed by the reference measurement it derives from; per-chunk growth accounts for the non-rolling KV-cache path retained in v1 (§3.5 decision 3).

Wall-time / throughput (reference DiT configuration):

• Per-chunk DiT denoising ≤ 8.5 s for the first 7 chunks (covers up to frame_num=81). Reference: chunks 1–7 measured at 6.2, 6.3, 6.6, 6.9, 7.3, 7.6, 8.0 s — average 7.4 s, maximum 8.0 s. Growth per chunk is monotonic and reflects the KV cache size carried forward; the target's 8.5 s ceiling bounds the worst-case chunk in this range with a modest margin.
• End-to-end DiT denoising ≤ 55 s at frame_num=81 (7 chunks). Reference measurement: 51.94 s FA2, 53 s SDPA-fallback — the two backends are at parity within 2%.
• End-to-end wall time ≤ 250 s at frame_num=81, excluding the one-time per-process setup cost (T5 / VAE / first DiT checkpoint load, which depend on disk and are not part of the per-request budget). Reference: 242 s measured for a full first-request run including setup; warm per-request budget will be reported in the PR description once the test-harness pattern is settled.

Memory targets (same reference point):

• Peak VRAM ≤ 70 GiB at frame_num=81 (reference: 67.4 GiB measured on the 04-22 SDPA baseline at identical shape; FA2 does not materially change peak memory).
• Peak VRAM ≤ 50 GiB at frame_num=25 (estimated from the frame-count scaling).

4. Correctness & Testing Plans

Following docs/contributing/ci/CI_5levels.md. Five-level test system; v1 targets L1 + L2 + L3. L4 (nightly perf, doc example tests) is nice-to-have; L5 (stability / reliability) is follow-up.

4.1 Invariants to check

• Plücker embedding parity. Our in-tree get_plucker_embeddings output matches upstream wan/utils/cam_utils.py:get_plucker_embeddings bit-for-bit on fixed (intrinsics, poses) input. Basis case for all downstream equivalence.
• Cache-update invariance. After a chunk's cache-update pass, the self-KV cache state equals the state obtained by a full forward over the clean x0 at t=0, within BF16 numerical tolerance. Tests this on a single chunk with fabricated timesteps.
• Chunk boundary continuity. The current_start RoPE offset in chunk N+1 aligns with the trailing position of chunk N's written cache entries (global_end_index).
• CFG-parallel cache separation. _pos and _neg KV caches do not share storage; mutating one does not mutate the other.
• Protocol round-trip. An infer payload with known-good cameras: {intrinsics, poses} decodes server-side (through the msgpack numpy codec) to the same float32 tensors the client produced. Handshake / error / reset framings match the documented shapes.
• Session-id change resets cleanly. Within a single WebSocket connection, switching session_id (without an explicit reset RPC) causes the next infer to enter with a clean state container — KV caches re-initialised, frame buffer empty, RoPE offset zero, language re-encoded — with no carryover from the previous session. Mirrors PR #2162's serving-layer change-detector behaviour (openpi_serving.py:120-125). (Concurrent-session isolation across session_ids is out of v1 scope; the v1 server runs one session at a time — see §3.4 session semantics.)

4.2 L1 — Unit tests (CPU, <15 min, PR-ready label, @pytest.mark.core_model)

Files under tests/diffusion/models/lingbot_world_fast/:

• test_plucker.py — Plücker embedding equivalence vs. upstream reference on synthetic (K, poses). Includes edge cases (identity pose, pure translation, pure rotation).
• test_session_state.py — LingBotWorldFastState allocation shapes, index arithmetic (local_end_index, global_end_index), rolling eviction path gated off, reset semantics (session-id change, prompt change), CFG _pos/_neg cache separation.
• test_protocol_validation.py — msgpack-encoded infer / reset payload validation: missing required fields, malformed cameras arrays, bad dtypes, session-id churn. Asserts error-frame responses (not connection-close).
• test_schedule.py — timestep-index selection against FlowUniPCMultistepScheduler produces the expected 4 steps.

4.3 L2 — Basic E2E (GPU, <15 min, PR-ready, @pytest.mark.core_model)

• tests/e2e/offline_inference/test_lingbot_world_fast.py — load the Fast pipeline with a dummy / mock-weight model; drive a 2-chunk session through the pipeline (bypassing the WebSocket layer), asserting state-container persistence between chunks, output shape (chunk_latent_frames, 16, lat_h, lat_w) per chunk, no NaN / Inf.
• tests/e2e/online_serving/test_lingbot_world_fast.py — start a minimal realtime server; open a WebSocket to /v1/realtime/world/camera; assert handshake returns a well-formed CameraServerConfig; send one infer with dummy image + prompt + cameras; assert a frame response with non-empty mp4 payload (content ignored; envelope / framing / msgpack shape checked). Repeat with a reset and one more infer to exercise the state-clear path.

4.4 L3 — Real-model integration, accuracy, performance (GPU, <30 min, post-merge, @pytest.mark.advanced_model)

• tests/e2e/offline_inference/test_lingbot_world_fast_expansion.py:
  • Load real Fast weights from HF cache (fixture skip if unavailable).
  • Drive the pipeline directly through 9 chunks with upstream's run_fast.sh case2 inputs (examples/04 image + 533-pose trajectory sliced into 9 chunks + Great-Wall prompt, seed 42, 480×832), reassemble a 25-frame video.
  • Assert peak VRAM ≤ 50 GiB and end-to-end DiT denoising ≤ 20 s per §3.8 (reference: 12.88 s measured at frame_num=25).
  • Structural-similarity (SSIM) of the first and last output frame vs. a stored golden tensor ≥ 0.85 (perceptual check, not bit-parity — accommodates 4-step stochastic sampling).
• tests/e2e/online_serving/test_lingbot_world_fast_expansion.py:
  • Same inputs, driven through the /v1/realtime/world/camera WebSocket endpoint as 9 sequential infer requests. Reassemble returned MP4 fragments; apply the same SSIM check. Additionally verify: session-id churn resets state correctly; a mid-session reset produces a new first-chunk-initialisation on the following infer; connection-close drops server-side state (observable by asserting pipeline.state.current_start_frame == 0 and empty KV caches after the disconnect handler runs).
• Both marked @pytest.mark.advanced_model @pytest.mark.core_model (the basic run and deep-validation split per the CI doc's §2.4.1 guidance).

4.5 L4 / L5 — deferred

• L4 functional expansion — 720p, longer frame counts (frame_num=125), alternate prompts, CFG on/off. Nightly cadence once L3 is landed.
• L4 doc-example tests — tests/example/online_serving/test_lingbot_world_fast.py that executes the curl + Python examples from examples/online_serving/video_generation/ and asserts successful completion. Added when the examples land.
• L4 perf tests — tests/dfx/perf/tests/test.json entry with the reference config + target throughput / latency thresholds derived from §3.8. Added when the in-tree perf harness matures around diffusion workloads.
• L5 stability — long-running WebSocket session with repeated chunk requests (and periodic reset + new-session churn); watch for KV cache / frame-buffer memory growth or state leakage across sessions on the same connection. Follow-up.

6. References

• RFC #1987 — World Model Support (umbrella RFC).
• PR #2162 — [WIP][Diffusion] DreamZero world model integration (direct precedent for both the realtime WebSocket serving shape in §3.4 and the session-scoped state container pattern in §3.5 decision 2).
• PR #2885 — [WIP][Diffusion] LingBot-VA world-action model support (robotics-control sibling).
• Issue #3072 — this RFC's own thread; 2026-04-23 review from @TKONIY directed the alignment with PR #2162's realtime API and stage-manager KV-cache pattern.
• Upstream repo: https://github.com/Robbyant/lingbot-world
• Paper (LingBot-World): arXiv:2601.20540.
  • Figure 5 — pipeline diagram showing "Action Scale & Shift with Plücker Encoder" as the per-block conditioning mechanism.
  • README Model Download table — canonical variant / control-signal mapping (Base Cam / Base Act / Fast).
• Paper (LingBot-VA): arXiv:2601.21998 (sibling model, PR #2885).
• In-tree testing framework: docs/contributing/ci/CI_5levels.md — multi-level test system (L1-L5).
• In-tree adjacent precedents: vllm_omni/diffusion/models/dreamzero/state_dreamzero.py (session-scoped state container — the direct precedent for §3.5 decision 2 and §3.4 API shape, via PR #2162); vllm_omni/entrypoints/openai/realtime/robot/{openpi_connection,openpi_serving}.py (realtime-WebSocket serving handler — the direct precedent for §3.4, via PR #2162); vllm_omni/diffusion/models/glm_image/glm_image_transformer.py (module-resident KV cache — earlier in-tree KV-cache precedent); vllm_omni/diffusion/models/wan2_2/wan2_2_transformer.py (Wan 2.2 primitives imported).
• Camera representation: Plücker ray embedding convention from CameraCtrl (arXiv 2404.02101 §3.2); field shape follows Nerfstudio / ViPE / COLMAP conventions.

Feedback Period.

One week

CC List.

@yinpeiqi @ywang96 @Gaohan123 @hsliuustc0106 @asukaqaq-s @IlyasFardaouix @matchyc @TKONIY @shanghuaye504-byte @kevinzeng08 @zhujiapeng @zzhang-fr

Any Other Things.

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.