[RFC]: Continuous Quantization Support

Quantization Matrix

Tracks which models are supported by each quantization method and which components are quantized.

✅ Quantized | ❌ Kept at full precision (intentionally skipped) | — N/A | 🙋 Help wanted | ⏳ PR raised

Category: 🖼️ Image | 🎬 Video | 🔊 Audio | 🌐 Omni — Type: D = Dynamic (online) | S = Static (offline, pre-quantized checkpoint required)

---

Contributing: Hardware Requirement

Before claiming a row, please comment with the GPU/accelerator you have access to for end-to-end validation. PRs without a testing device have stalled in review — please confirm hardware upfront so reviewers aren't left validating work the author can't run.

Rough requirements by method:

• FP8 D (dynamic online) — Hopper (H100 / H200), Ada (L40S / RTX 4090), or Blackwell (B200 / RTX 5090). FP8 tensor cores need sm_89+.
• NvFP4 S / MXFP4 — Blackwell only (sm_100).
• INT8 D — any NVIDIA GPU or Ascend 910B2.
• GGUF S / BitsandBytes W4 / AutoRound W4A16 — any NVIDIA; 24 GiB+ recommended for larger DiTs.

Video models (Wan 2.2, HunyuanVideo-1.5, LTX-2, MOVA, etc.) realistically need 80 GiB VRAM (A100 80GB / H100 / H200) for full end-to-end tests; 48 GiB (L40S / RTX 6000 Ada) is the practical floor with CPU offload + VAE tiling.

When claiming a task, please comment with:

1. Model + method (e.g. "DreamID-Omni FP8 D")
2. GPU / accelerator + VRAM
3. (optional) TP size and expected timeline

If you have any questions and not sure if your thoughts/contributions align with our roadmaps. The fastest way to reach out to me is via Wechat. My ID is LSY2717785144

---

Cat	Model	Method	Type	Platform	Checkpoint	Attn	MLP	MoE	KV Cache	Notes
🌐	Qwen3-Omni	FP8	D	CUDA	—	✅#1764	✅#1764	✅#1764	🙋	Thinker only; talker/audio/visual BF16
🌐	Qwen3-Omni	ModelOpt FP8	S	CUDA	asdazd/Qwen3-Omni-...-FP8	✅#1764	✅#1764	✅#1764	✅#1764	Thinker LLM only; lm_head+mlp.gate excluded
🌐	Qwen3-Omni	NvFP4	S	Blackwell	shunyang90/...-NVFP4	✅#4025	✅#4025	✅#4025	🙋	Thinker NVFP4 W4A4 on Blackwell; lm_head excluded via #4528. Encoders/talker BF16
🌐	Qwen3-Omni	AutoRound W4A16	S	CUDA	Intel/Qwen3-Omni-30B-A3B-Instruct-int4-AutoRound	✅#2670	✅#2670	✅#2670	—	Thinker only; 68% size reduction (66→21 GiB); omni_bench acc -0.02 vs BF16
🌐	Qwen2.5-Omni	FP8	D	CUDA	—	🙋	🙋	—	🙋	@wuli666; L40S
🌐	Qwen2.5-Omni	AutoRound W4A16	S	CUDA	Intel/Qwen2.5-Omni-7B-int4-AutoRound	✅#2670	✅#2670	—	—	48% size reduction (21→11 GiB)
🎬	Wan 2.2	FP8	D	CUDA	—	🙋	🙋	—	—	@ArtificialRay; blocked on #2920 (quant_config wiring)
🎬	Wan 2.2	ModelOpt FP8	S	CUDA	—	✅#3305	✅#3305	—	—	Phase 1 video-gen; tracks #2709
🎬	Wan 2.2	MXFP4	S	Blackwell	—	⏳#3053	⏳#3053	—	—	@Liziqi-77; Marlin W4A16 kernel
🎬	Wan 2.2	MXFP4 W4A4	D/S	Ascend	—	✅#3578	✅#3578	—	—	vLLM-Ascend NPU; online + dual-scale offline (T2V/I2V)
🎬	Wan 2.2	MXFP8 W8A8	D/S	Ascend	—	✅#3140	✅#3140	—	—	vLLM-Ascend NPU (T2V/I2V/TI2V)
🎬	Wan 2.2	AutoRound W4A16	S	CUDA	—	✅#3353	✅#3353	—	—
🎬	Wan 2.2	GGUF	S	CUDA	QuantStack/Wan2.2-T2V-A14B-GGUF	🙋	🙋	—	—	Draft #1756 closed; new volunteer welcome
🎬	Helios	FP8	D	CUDA	—	✅#1916	✅#1916	—	—
🎬	HunyuanVideo-1.5	FP8	D	CUDA	—	✅#1516	✅#1516	—	—
🎬	HunyuanVideo-1.5	ModelOpt FP8	S	CUDA	—	✅#3305	✅#3305	—	—	Phase 1 video-gen; I2V transformer wired in #4245
🎬	LTX-2	FP8	D	CUDA	—	✅#3700	✅#3700	—	—	Online fp8/int8
🎬	MOVA	FP8	D	CUDA	—	🙋	🙋	—	—
🔊	Stable-Audio-Open	FP8	D	CUDA	—	⏳#3524	⏳#3524	—	—	@akshatvishu; L4 24GB
🔊	Voxtral TTS	FP8	D	CUDA	—	✅#3036	✅#3036	—	—	@akshatvishu
🔊	F5-TTS	FP8	D	CUDA	—	🙋	🙋	—	—
🖼️	Z-Image	FP8	D	CUDA	—	✅#1034	✅#1034	—	—
🖼️	Z-Image	GGUF	S	CUDA	unsloth/Z-Image-Turbo-GGUF	✅#1285	✅#1285	—	—
🖼️	Z-Image	INT8	D	Ascend	—	✅#1470	✅#1470	—	—
🖼️	Z-Image	BitsandBytes W4	D	CUDA	—	🙋	🙋	—	—	Peak VRAM 24.5→17.1 GiB on Z-Image-Turbo; draft #1528 closed, new volunteer welcome
🖼️	Z-Image	Nunchaku SVDQuant W4A4	S	Blackwell	ultranationalism/nunchaku-z-image-turbo	⏳#3830	⏳#3830	—	—	~2.2x speedup vs BF16 on RTX 5090 (#1986 superseded)
🖼️	Qwen-Image	FP8	D	CUDA	—	✅#1034	✅#1034	—	—	img_mlp skip recommended
🖼️	Qwen-Image	GGUF	S	CUDA	QuantStack/Qwen-Image-GGUF	✅#1755	✅#1755	—	—
🖼️	Qwen-Image	INT8	D	Ascend	—	✅#1470	✅#1470	—	—
🖼️	Qwen-Image	AutoRound W4A16	S	CUDA	—	✅#3588	✅#3588	—	—
🖼️	Qwen-Image-2512	FP8	D	CUDA	—	✅#1034	✅#1034	—	—	Same arch as Qwen-Image
🖼️	Qwen-Image-Edit	FP8	D	CUDA	—	✅#1034	✅#1034	—	—	Same arch as Qwen-Image (#2608 closed)
🖼️	Qwen-Image-Edit-2509	FP8	D	CUDA	—	✅#1034	✅#1034	—	—	Same arch as Qwen-Image
🖼️	Qwen-Image-Layered	FP8	D	CUDA	—	✅#1034	✅#1034	—	—	Same arch as Qwen-Image
🖼️	Bagel	FP8	D	CUDA	—	✅#1764	✅#1764	—	—	MoT dual-path (understand+gen) both quantized
🖼️	FLUX.1-dev	FP8	D	CUDA	—	✅#1640	✅#1640	—	—
🖼️	FLUX.1-dev	Auto-Round	S	CUDA / Intel	vllm-project-org/FLUX.1-dev-AutoRound-w4a16	✅#1777	✅#1777	—	—	W4A16; RFC #1325
🖼️	FLUX.1-Kontext	FP8	D	CUDA	—	✅#2184	✅#2184	—	—
🖼️	FLUX.2-klein	GGUF	S	CUDA	unsloth/FLUX.2-klein-4B-GGUF	✅#1285	✅#1285	—	—
🖼️	FLUX.2-dev	FP8	D	CUDA	—	⏳#3027	⏳#3027	—	—	@Songrui625; #1806 requested
🖼️	FLUX.2-dev	NvFP4	S	Blackwell	black-forest-labs/FLUX.2-dev-NVFP4	🙋	🙋	—	—	comfy-kitchen kernel path; draft #2517 closed, new volunteer welcome
🖼️	HunyuanImage3	FP8	D	CUDA	—	✅#1935	✅#1935	✅#1935	—	MoE router skipped (hardcoded)
🖼️	HunyuanImage3	Offline FP8	S	Ascend	—	✅#2979	✅#2979	✅#2979	—	@jiangmengyu18; vLLM-Ascend offline path
🖼️	GLM-Image	FP8	D	CUDA	—	✅#2292	✅#2292	—	—
🖼️	GLM-Image	AutoRound W4A16	S	CUDA	—	✅#3059	✅#3059	—	—	@lvliang-intel; extends AutoRound infra to GLM-Image
🖼️	OmniGen2	FP8	D	CUDA	—	✅#2441	✅#2441	—	—	@zhangj1an; pads hidden_size=2520 to mult-of-16 for CUTLASS fast path
🖼️	StableDiffusion3	FP8	D	CUDA	—	⏳#3061	⏳#3061	—	—	@hongzhi-gao; down_proj MLP only, attn/context BF16
🖼️	Stable-Diffusion3.5	FP8	D	CUDA	—	⏳#3061	⏳#3061	—	—	Shared pipeline with SD3
🖼️	DreamID-Omni	FP8	D	CUDA	—	✅#3902	✅#3902	—	—	Online fp8/int8
🖼️	MammothModa2	FP8	D	CUDA	—	🙋	🙋	—	—
🖼️	LongCat-Image	FP8	D	CUDA	—	⏳#2633	⏳#2633	—	—	@lcukyfuture
🖼️	LongCat-Image-Edit	FP8	D	CUDA	—	⏳#2633	⏳#2633	—	—	Same arch as LongCat-Image
🖼️	Ovis-Image	FP8	D	CUDA	—	🙋	🙋	—	—	@yangjianjuan; L20
🖼️	Magihuman	FP8	D	CUDA	—	🙋	🙋	—	—
🖼️	NextStep-1.1	FP8	D	CUDA	—	⏳#2482	⏳#2482	—	—	@banparth; tracking #1815
🖼️	Diffusion (all DiT)	MXFP4 W4A16	D	CUDA	—	🙋	🙋	—	—	Marlin W4A16 kernel; ~3.8x weight compression; generic draft #2200 closed (per-model MXFP4 tracked separately)

Notes:

1. VAE and Audio Codec/Vocoder are precision-sensitive and kept at full precision for all current implementations. Text encoder online FP8 quantization landed for Z-Image in #1338; tracking the same pattern for other models in the Text Encoder Online Quantization section below.
2. KV Cache quantization applies to omni models only (LLM thinker has autoregressive KV cache). Diffusion-only models do not use KV cache.
3. Omni models: dynamic FP8 scopes to thinker/LLM only; talker, audio encoder, vision encoder, and code2wav stay BF16.
4. MoE Router is always kept at full precision — hardcoded quant_config=None in HunyuanImage3, documented as sensitive for NvFP4.
5. ModelOpt framework: #2709 tracks the umbrella ModelOpt-based offline FP8 path. #2913 (merged) covers Phase 1 for image-gen (Flux, Flux2-Klein, Qwen-Image, HunyuanImage-3); #3305 (merged) adds the video-gen counterpart (HunyuanVideo-1.5, Wan2.2), with #3570 extending to Phase 2 mixed FP8/NVFP4 for image-gen and #4155 enabling ModelOpt FP8 under batched diffusion serving.
6. For the broader Model × Feature acceleration matrix (TeaCache, SP, TP, CPU offload, etc.), see #1217.

---

Text Encoder Online Quantization

Following #1338 (which enabled online FP8 for the Z-Image text encoder), the same pattern can be extended to other models that already have online FP8 working or in flight for the DiT. Help wanted.

Pattern reference (from #1338):

• vllm_omni/diffusion/models/utils.py — text encoder quantization helpers
• vllm_omni/diffusion/model_loader/diffusers_loader.py — loader integration
• vllm_omni/diffusion/models/z_image/pipeline_z_image.py — per-model wiring

Cat	Model	Method	Type	Platform	Status	Notes
🖼️	Z-Image	FP8	D	CUDA	✅#1338	Pattern reference
🖼️	Qwen-Image	FP8	D	CUDA	🙋	@lulugoodcoder; mirrors #1338
🖼️	Qwen-Image-2512	FP8	D	CUDA	🙋	Same arch as Qwen-Image
🖼️	Qwen-Image-Edit	FP8	D	CUDA	⏳#3484	@xRay2016 (H20 96GB); follows #1338 #3279
🖼️	Qwen-Image-Edit-2509	FP8	D	CUDA	⏳#3998
🖼️	Qwen-Image-Layered	FP8	D	CUDA	⏳#3998
🖼️	Bagel	FP8	D	CUDA	🙋
🖼️	FLUX.1-dev	FP8	D	CUDA	🙋	@leohuang257 (RTX 5090); dual T5-XXL + CLIP-L
🖼️	FLUX.1-Kontext	FP8	D	CUDA	🙋
🖼️	FLUX.2-dev	FP8	D	CUDA	🙋	DiT FP8 in flight via #3027
🖼️	HunyuanImage3	FP8	D	CUDA	🙋
🖼️	GLM-Image	FP8	D	CUDA	🙋
🖼️	OmniGen2	FP8	D	CUDA	🙋
🖼️	StableDiffusion3	FP8	D	CUDA	🙋	DiT FP8 in flight via #3061
🖼️	Stable-Diffusion3.5	FP8	D	CUDA	🙋	Shared pipeline with SD3
🖼️	LongCat-Image	FP8	D	CUDA	🙋	DiT FP8 in flight via #2633
🖼️	LongCat-Image-Edit	FP8	D	CUDA	🙋	Same arch as LongCat-Image
🖼️	Ovis-Image	FP8	D	CUDA	🙋
🖼️	NextStep-1.1	FP8	D	CUDA	🙋	DiT FP8 in flight via #2482
🖼️	DreamID-Omni	FP8	D	CUDA	🙋
🖼️	MammothModa2	FP8	D	CUDA	🙋
🖼️	Magihuman	FP8	D	CUDA	🙋
🎬	Wan 2.2	FP8	D	CUDA	🙋
🎬	Helios	FP8	D	CUDA	🙋
🎬	HunyuanVideo-1.5	FP8	D	CUDA	🙋
🎬	LTX-2	FP8	D	CUDA	🙋	DiT FP8 merged in #3700
🎬	MOVA	FP8	D	CUDA	🙋

---

CC List

@hsliuustc0106 @ZJY0516 @SamitHuang @wtomin @mxuax @gcanlin @dongbo910220 @Isotr0py @zzhuoxin1508 @alex-jw-brooks @RuixiangMa @yiliu30 @yjb767868009 @faaany @zsl24 @david6666666 @zhiming33416 @xuechendi @lengrongfu @godnight @ultranationalism @akshatvishu @linyueqian @Sy0307 @lyj-jjj @Fishermanykx @zhudianGG @lvliang-intel @vraiti @banparth @erfgss @Michael-Zzq @Songrui625 @yangjianjuan @hongzhi-gao @JasonJ2021 @zhangj1an @wuli666 @lcukyfuture @ArtificialRay @Liziqi-77 @jiangmengyu18