Repository metrics

Stars: (4,990 stars)
PR merge metrics: (PR metrics pending)

Description

Motivation.

To improve the usability and inference speed of the Omni model, we propose a roadmap for the Omni model. The main models and features we are targeting are as follows:

Model	Stage	Batch	Cuda Graph	Async Chunk	Streaming input	Streaming output	Prefix cache	Chunked Prefill	Quantization
Qwen3-Omni	✅	✅	✅	✅		✅			✅
Qwen2.5-Omni	✅	✅

Proposed Change.

1. Qwen3-Omni

Latest performance data(2026-03-22):

test_name	max_concurrency	mean_e2el_ms	mean_audio_ttfp_ms	mean_audio_rtf
qwen3_omni	1	4887.274765	4780.586555	0.168678568
qwen3_omni	4	5515.413707	5411.484435	0.20502113
qwen3_omni	10	9055.343315	8951.584469	0.335526352
qwen3_omni_chunk	1	4132.492401	396.5581684	0.149085402
qwen3_omni_chunk	4	5871.904884	872.6258571	0.212834243
qwen3_omni_chunk	10	12278.46819	2582.148682	0.454904836

Feature list

Feature	Title	Author	RFC	PR	Status
Stage	add support for Qwen3-omni	R2-Y	https://github.com/vllm-project/vllm-omni/pull/55	https://github.com/vllm-project/vllm-omni/pull/55	Merged✅
Batching	Support online batch inference	ZeldaHuang	https://github.com/vllm-project/vllm-omni/issues/410 https://github.com/vllm-project/vllm-omni/issues/420	https://github.com/vllm-project/vllm-omni/pull/438 https://github.com/vllm-project/vllm-omni/issues/1211	Merged✅
Cuda Graph	Qwen3 Omni support cuda graph	ZeldaHuang	https://github.com/vllm-project/vllm-omni/issues/421	https://github.com/vllm-project/vllm-omni/pull/523 https://github.com/vllm-project/vllm-omni/pull/669	Merged✅
Async Chunk	Support async computation and communication across stages by chunks	amy-why-3459 R2-Y Shirley125	https://github.com/vllm-project/vllm-omni/issues/268	https://github.com/vllm-project/vllm-omni/pull/727 https://github.com/vllm-project/vllm-omni/pull/951	Merged✅
Streaming input	Qwen3-Omni supports streaming input	Shirley125	https://github.com/vllm-project/vllm-omni/issues/1951		In progress
Streaming output	Support async computation and communication across stages by chunks	amy-why-3459	https://github.com/vllm-project/vllm-omni/issues/268	https://github.com/vllm-project/vllm-omni/pull/727	Merged✅
Prefix cache	Enable Prefix Caching with Hidden-State I/O (Multi-round / Service Scenarios)		https://github.com/vllm-project/vllm-omni/issues/1184		In progress
Chunked Prefill	Support chunked prefill	R2-Y	https://github.com/vllm-project/vllm-omni/issues/948		In progress
Quantization	Improving FP8 Support for Qwen3-Omni in vLLM-Omni	lishunyang12	https://github.com/vllm-project/vllm-omni/issues/1544	https://github.com/vllm-project/vllm-omni/pull/1764	Merged✅

Performance

Title	Author	RFC	PR	Status
Qwen3-Omni performance optimization	amy-why-3459	https://github.com/vllm-project/vllm-omni/issues/1377	https://github.com/vllm-project/vllm-omni/pull/1378	Merged✅
Reduce inter-packet latency in async chunk	ZeldaHuang	/	https://github.com/vllm-project/vllm-omni/pull/1656	Merged✅
Qwen3-omni: code predictor with re-prefill + SDPA and eliminate decode hot-path CPU round-trips	LJH-LBJ	/	https://github.com/vllm-project/vllm-omni/pull/1758	Merged✅
Fix slow hasattr in CUDAGraphWrapper.getattr	ZeldaHuang	/	https://github.com/vllm-project/vllm-omni/pull/1982	Merged✅

BugFix

Title	Author	RFC	PR	Status
Fix qwen3-omni async thinker to talker decode alignment for #1758	Sy0307	https://github.com/vllm-project/vllm-omni/issues/1830	https://github.com/vllm-project/vllm-omni/pull/2019	Merged✅
Fix memory leak: missing chunk_transfer_adapter.cleanup() in OmniARScheduler	dubin555	/	https://github.com/vllm-project/vllm-omni/pull/2028	Merged✅
support to change the speaker of qwen3-omni	R2-Y	https://github.com/vllm-project/vllm-omni/issues/1845	https://github.com/vllm-project/vllm-omni/pull/1963	Merged✅

2. Qwen2.5-Omni

p0:

Feedback Period.

No response

CC List.

@hsliuustc0106 @Gaohan123 @R2-Y @Shirley125 @ZeldaHuang @tzhouam @linyueqian No response

Any Other Things.