[RFC]: Low-Precision Quantization (MXFP4) Support for Multimodal Models in vLLM-Omni

Related Issues

1057

Summary

This RFC proposes the implementation of low-precision quantization algorithms (MXFP4) and advanced quantization methods (FourOverSix, Level-2 quantization algorithm, etc.) for multimodal models in the vLLM-Omni framework. We plan to start with Wan2.2 as the initial target model, with the goal of significantly reducing memory footprint and improving inference efficiency while maintaining model accuracy.

Motivation

Background

Multimodal AI models like Wan2.2, which support text-to-video generation and other multimodal tasks, require substantial GPU memory for inference. This creates significant barriers for deployment, especially for organizations with limited computational resources.

Current State

Now, some work has already been done on FP8 and INT8 in the community. However, there's a gap in support for:

1. Advanced 4-bit Floating-Point Formats: MXFP4 offer superior accuracy-efficiency trade-offs compared to INT4
2. Multimodal-Specific Quantization: Different modalities may require different quantization strategies
3. Emerging Quantization Algorithms: Methods like FourOverSix show promising results but are not yet integrated

Value Proposition

Implementing these quantization techniques will:

• Reduce memory usage by 60-75%
• Improve inference throughput
• Maintain model quality within 1-2% of original models
• Democratize access to powerful multimodal AI capabilities

Proposed Planning

Phase 1: Initial Model Support

Target Model: Wan2.2 (likely to be the first model)

Rationale for Wan2.2:

• Large parameter size (14B) benefits significantly from quantization
• Active community interest and usage
• Representative DiT architecture for multimodal generation

Quantization Methods to Explore:

MXFP4 (Microsoft's Microscaling Format)

• Format: 32 values share one scaling factor
• Advantages: Optimized for memory bandwidth, suitable for transformers
• Challenges: Coarse granularity may impact accuracy

Components to Consider:

• Vision encoder
• Text encoder
• Diffusion Transformer (DiT)
• Temporal layers

Phase 2: Advanced Quantization Algorithms

Methods to Investigate:

FourOverSix

• Concept: Adaptive bit allocation (4-6 bits) based on layer sensitivity
• Advantages: Near-lossless compression with error compensation

Enhanced AWQ for Multimodal Models

• Concept: Modality-aware quantization strategies
• Advantages: Protects critical channels for each modality

Other Community-Recommended Algorithms

• Open to suggestions from the community
• Will evaluate based on accuracy, performance, and implementation complexity

Phase 3: Framework Integration

Key Components:

• Modular quantizer architecture (extending existing FP8/INT8 framework)
• User-friendly API design (following #896 interface design)
• Configuration system (YAML + Python SDK)
• Documentation and examples

Integration Points:

• Build upon existing vllm_omni/diffusion/quantization/ framework
• Extend DiffusionQuantizationConfig base class
• Add new quantization methods to the registry

Phase 4: Model Expansion

After successful implementation on Wan2.2, we plan to extend support to:

• Qwen2.5-Omni series
• Qwen3-Omni series
• Other multimodal models based on community needs

Next Steps

The detailed technical design, implementation timeline, testing strategy, and validation metrics will be provided in future RFCs after thorough research and community feedback.

Immediate Actions

1. Community Discussion: Gather feedback on target models and quantization methods
2. Detailed Design: Prepare comprehensive technical specifications for approved methods

Conclusion

This RFC outlines our intention to implement advanced low-precision quantization support for multimodal models in vLLM-Omni. We believe this work will significantly improve the accessibility and efficiency of multimodal AI deployment. We look forward to collaborating with the community to refine this proposal and develop a comprehensive implementation plan.

Proposed Change.

Please provide the detailed design document of the RFC using the template.

Feedback Period.

No response

CC List.

No response

Any Other Things.

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.