[New Model]: Audio-Omni

The model to consider.

Audio-Omni — an end-to-end framework that unifies audio understanding, generation, and editing across general sound, music, and speech.

• Code: https://github.com/ZeyueT/Audio-Omni
• Weights: https://huggingface.co/HKUSTAudio/Audio-Omni
• Paper: Audio-Omni: Extending Multi-modal Understanding to Versatile Audio Generation and Editing (SIGGRAPH 2026), arXiv:2604.10708
• License: code CC BY-NC 4.0; weights research-use-only (commercial use needs author authorization)

Capabilities

• Generation: Text-to-Speech (with voice cloning), Voice Conversion, Text-to-Audio (T2A), Text-to-Music (T2M), Video-to-Audio (V2A), Video-to-Music (V2M)
• Editing: add / remove / extract / style-transfer on existing audio
• Understanding: audio/video question answering

Architecture (this is the key reason it fits vllm-omni well)

• Backbone: a frozen Qwen2.5-Omni MLLM for reasoning / conditioning
• Synthesizer: a trainable Diffusion Transformer that decodes to waveform
• Aux: a Synchformer checkpoint used only for the V2A / V2M tasks

Released artifacts: Audio-Omni.json (config), model.ckpt (DiT + adapters), synchformer_state_dict.pth.

The closest model vllm-omni already supports.

Two existing models cover the two halves of Audio-Omni almost exactly:

1. Qwen2.5-Omni (Qwen2_5OmniForConditionalGeneration, already in registry.py, including the Qwen2_5OmniToken2WavDiTModel diffusion module). Audio-Omni's reasoning backbone is Qwen2.5-Omni and is frozen, so the base LM can be reused as-is under vLLM PagedAttention.
2. VoxCPM2 — the reference for the "vLLM-native base LM + diffusion side-computation" pattern: the AR base model runs under PagedAttention while the diffusion / VAE synthesis runs outside vLLM and is attached via the runner post-processing hook (see plan/voxcpm2_native_ar_design.md). Audio-Omni's frozen-MLLM + trainable-DiT split is the same shape.

So the integration is mostly composition of patterns we already have rather than net-new architecture.

What's your difficulty of supporting the model you want?

Medium overall, but it factors cleanly into beginner-friendly milestones. The recommendation is not to support all six generation tasks + editing + understanding at once. Scope the first PR to the speech path:

Milestone 1 (good first issue): Text-to-Speech + voice cloning, offline.

• Reuse the frozen Qwen2.5-Omni thinker as the base LM (already supported).
• Attach the Diffusion Transformer as a side-computation via the runner post-processing hook (VoxCPM2 pattern).
• Get end2end.py producing audio that matches the reference checkpoint.

Milestone 2: online serving (/v1/audio/speech) with the standard 5-point serving_speech.py wiring + voice-cloning prompt path.

Milestone 3+ (follow-ups, not for the first PR): T2A / T2M, then V2A / V2M (requires Synchformer + video preprocessing), then editing and understanding.

Known difficulties to flag for whoever picks this up:

• Diffusion sampler is a new hot path — CFM/DiT solver steps must follow the hot-loop GPU discipline (no .item()/.cpu() inside the solver; float32 for the decoder; profile before optimizing).
• Multimodal preprocessing — Audio-Omni accepts text + optional reference audio (voice clone) + optional video (V2A/V2M, deferred). Only Milestone 1's text + ref-audio path is needed first.
• Weights are research-only / CC BY-NC 4.0 — fine for an opt-in model integration, but the PR should not vendor weights; load from HKUSTAudio/Audio-Omni.
• Checkpoint is a single model.ckpt (not HF safetensors sharded layout) — needs a small custom weight loader.

Contributor guides for each step:

• TTS integration workflow: docs/contributing/model/adding_tts_model.md
• Diffusion-side-path reference: VoxCPM2 (plan/voxcpm2_native_ar_design.md) and the in-tree Qwen2.5-Omni Token2Wav DiT.

Use case and motivation

Audio-Omni is, to date, the first single model to unify audio understanding + generation + editing across sound, music, and speech, with instruction-following emergent from its MLLM backbone. Supporting it in vllm-omni gives the community a high-throughput serving path for an instruction-driven "any audio task" model, and — because the backbone (Qwen2.5-Omni) and the diffusion-side-path mechanics (VoxCPM2) already exist in tree — it is an excellent on-ramp for a new contributor to learn the full TTS/diffusion integration flow end-to-end on a model that is mostly assembling existing primitives.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.