vllm-project/vllm-omni

[RFC]: Unify example model usage — migrate all special-usage examples to standard task examples + declarative model_extras

Open

#4,539 opened on Jun 18, 2026

View on GitHub
 (0 comments) (1 reaction) (1 assignee)Python (1,067 forks)github user discovery
criticalgood first issuehelp wantedhigh priority

Repository metrics

Stars
 (4,990 stars)
PR merge metrics
 (PR metrics pending)

Description

Motivation

PR #4225 introduced a unified parameter-declaration contract for DiT pipelines: per-model request/response params are declared centrally under vllm_omni/model_extras/<model>.py (registered in registry.py) via EXTRA_BODY_PARAMS / EXTRA_OUTPUT_PARAMS, routed into the engine through extra_body / custom_output, and the model's bespoke example script is deleted in favor of the shared task examples (text_to_image, image_to_image, …).

#4225 migrated BagelPipeline and SenseNovaU1Pipeline as the reference. Every other model still ships its own special-usage example directory, which causes:

  • New users must learn a different invocation per model instead of one standard task-oriented entrypoint.
  • Model-specific knobs are hardcoded in scattered scripts instead of declared once and validated centrally.
  • Example scripts drift from the engine API and are frequently untested.

Goal: migrate every remaining special-usage model example onto the standard task examples + a model_extras declaration, until no model requires bespoke example code.

Proposed Change

For each model in Sub-tasks, the migration mirrors #4225:

  1. Declare the model's extra request/response params in a new vllm_omni/model_extras/<model>.py and register it in vllm_omni/model_extras/registry.py.
  2. Route the pipeline's previously-hardcoded knobs through extra_body / custom_output (keep a legacy fallback during the transition).
  3. Delete the dedicated example dir(s) under examples/offline_inference/<model>/ and examples/online_serving/<model>/, folding usage into the relevant shared task example (text_to_image, image_to_image, image_to_video, text_to_video, text_to_audio, text_to_speech, …) — adding only an extra_body recipe where the model needs non-default params.
  4. Update e2e tests so the model is exercised through the standard path (and remove tests tied to the deleted bespoke script).
  5. Update markdown/README — remove the per-model README and document the model's extra_body knobs in the relevant task README / recipe.

⚠️ Contribution requirements (mandatory for every sub-task PR)

A sub-task PR will not be merged unless it includes all of the following — this is the single most important rule of this RFC:

  1. A complete, self-contained, reproducible execution script — the exact command(s) anyone can copy-paste to run the model through the standard task example, including the model id, all extra_body params, and any env vars. No "left as an exercise" snippets.
  2. The actual run output / results pasted into the PR description — e.g. the generated image/video/audio (or a link), console logs, and for serving the request + response. Reviewers must be able to see it actually ran on the migrated path.
  3. Updated e2e test(s) that cover the model on the standard path and pass in CI (mark appropriately, e.g. @pytest.mark.core_model).
  4. Updated markdown/README — the per-model README is removed and the knobs are documented in the corresponding task README/recipe.

Sub-tasks

Reference (already migrated in #4225 — use as the template):

  • BagelPipelinevllm_omni/model_extras/bagel.py
  • SenseNovaU1Pipelinevllm_omni/model_extras/sensenova_u1.py

Remaining model-specific (DiT) example directories to migrate (one issue each):

  • audiox @zzehli
  • cosmos3 @leohuang257
  • dreamzero @MrDongsls
  • helios @princepride
  • hunyuan_image3
  • internvla_a1
  • lance
  • magi_human @princepride
  • mammothmodal2_preview @moisf56
  • ming_flash_omni @yuanheng-zhao
  • vace @ForestWisdom

Out of scope: AR / Omni / TTS models with dedicated examples (qwen2_5_omni, qwen3_omni, mimo_audio, step_audio2, …) — the DiT model_extras contract does not directly apply; and how-to dirs (custom_pipeline, diffusers_pipeline_adapter) — these are guides, not model examples.

Feedback Period

One week from posting.

Any Other Things

Keep the legacy fallback lookup (as #4225 did) during the transition so migrations can land incrementally. Once all sub-tasks are done, follow up by removing the legacy fallback and any remaining per-model special-casing in the serving layer.

Contributor guide