[RFC]: Unify example model usage — migrate all special-usage examples to standard task examples + declarative model_extras
#4,539 opened on Jun 18, 2026
Repository metrics
- Stars
- (4,990 stars)
- PR merge metrics
- (PR metrics pending)
Description
Motivation
PR #4225 introduced a unified parameter-declaration contract for DiT pipelines: per-model request/response params are declared centrally under vllm_omni/model_extras/<model>.py (registered in registry.py) via EXTRA_BODY_PARAMS / EXTRA_OUTPUT_PARAMS, routed into the engine through extra_body / custom_output, and the model's bespoke example script is deleted in favor of the shared task examples (text_to_image, image_to_image, …).
#4225 migrated BagelPipeline and SenseNovaU1Pipeline as the reference. Every other model still ships its own special-usage example directory, which causes:
- New users must learn a different invocation per model instead of one standard task-oriented entrypoint.
- Model-specific knobs are hardcoded in scattered scripts instead of declared once and validated centrally.
- Example scripts drift from the engine API and are frequently untested.
Goal: migrate every remaining special-usage model example onto the standard task examples + a model_extras declaration, until no model requires bespoke example code.
Proposed Change
For each model in Sub-tasks, the migration mirrors #4225:
- Declare the model's extra request/response params in a new
vllm_omni/model_extras/<model>.pyand register it invllm_omni/model_extras/registry.py. - Route the pipeline's previously-hardcoded knobs through
extra_body/custom_output(keep a legacy fallback during the transition). - Delete the dedicated example dir(s) under
examples/offline_inference/<model>/andexamples/online_serving/<model>/, folding usage into the relevant shared task example (text_to_image,image_to_image,image_to_video,text_to_video,text_to_audio,text_to_speech, …) — adding only anextra_bodyrecipe where the model needs non-default params. - Update e2e tests so the model is exercised through the standard path (and remove tests tied to the deleted bespoke script).
- Update markdown/README — remove the per-model README and document the model's
extra_bodyknobs in the relevant task README / recipe.
⚠️ Contribution requirements (mandatory for every sub-task PR)
A sub-task PR will not be merged unless it includes all of the following — this is the single most important rule of this RFC:
- A complete, self-contained, reproducible execution script — the exact command(s) anyone can copy-paste to run the model through the standard task example, including the model id, all
extra_bodyparams, and any env vars. No "left as an exercise" snippets. - The actual run output / results pasted into the PR description — e.g. the generated image/video/audio (or a link), console logs, and for serving the request + response. Reviewers must be able to see it actually ran on the migrated path.
- Updated e2e test(s) that cover the model on the standard path and pass in CI (mark appropriately, e.g.
@pytest.mark.core_model). - Updated markdown/README — the per-model README is removed and the knobs are documented in the corresponding task README/recipe.
Sub-tasks
Reference (already migrated in #4225 — use as the template):
- BagelPipeline —
vllm_omni/model_extras/bagel.py - SenseNovaU1Pipeline —
vllm_omni/model_extras/sensenova_u1.py
Remaining model-specific (DiT) example directories to migrate (one issue each):
- audiox @zzehli
- cosmos3 @leohuang257
- dreamzero @MrDongsls
- helios @princepride
- hunyuan_image3
- internvla_a1
- lance
- magi_human @princepride
- mammothmodal2_preview @moisf56
- ming_flash_omni @yuanheng-zhao
- vace @ForestWisdom
Out of scope: AR / Omni / TTS models with dedicated examples (qwen2_5_omni, qwen3_omni, mimo_audio, step_audio2, …) — the DiT model_extras contract does not directly apply; and how-to dirs (custom_pipeline, diffusers_pipeline_adapter) — these are guides, not model examples.
Feedback Period
One week from posting.
Any Other Things
Keep the legacy fallback lookup (as #4225 did) during the transition so migrations can land incrementally. Once all sub-tasks are done, follow up by removing the legacy fallback and any remaining per-model special-casing in the serving layer.