[RFC]: Decouple vllm-omni GGUF quantization from upstream vllm#39583

Motivation.

Upstream vLLM has opened vllm-project/vllm#39583 proposing to deprecate and remove both bitsandbytes and GGUF quantization backends. The feedback period there is two weeks. If it lands as written, several user-facing diffusion features in vllm-omni break, so we need to agree on a plan before the upstream surface disappears.

Why the upstream RFC is fine for LLMs but a problem for Omni

The upstream motivation is reasonable on the LLM side: low usage (~0.5% bnb, ~0.1% GGUF), ~170 lines of conditional branches in linear.py / fused_moe/layer.py / vocab_parallel_embedding.py, neither backend migrated to weight_loader_v2, and ~6k lines of CUDA kernels in csrc/quantization/gguf/.

For vllm-omni, GGUF is not a legacy backend. It is a shipped, documented feature for diffusion transformers on consumer GPUs:

• Three model adapters: Z-Image, Flux2-Klein, Qwen-Image (plus Edit / Plus / Layered variants).
• Native --quantization gguf flag in examples/offline_inference/text_to_{image,video}/.
• Online: vllm serve ... --quantization-config '{"method":"gguf","gguf_model":"..."}'.
• CI functionality tests on L4: "TeaCache + Layerwise offload + GGUF", "CacheDiT + Ring=2 + HSDP=2 + VAE=2 + GGUF".
• Docs: docs/user_guide/diffusion/quantization/gguf.md, overview.md, FAQ roadmap entry.
• Community checkpoints (city96, QuantStack, unsloth) — diffusion users have no llama.cpp analogue to fall back to.

Critically, none of the upstream complaints apply to our GGUF path:

• We don't touch linear.py hot paths (no adjust_bitsandbytes_4bit_shard, no UninitializedParameter, no data_container).
• We don't go through fused_moe/layer.py.
• We don't use model_loader/gguf_loader.py or its meta-device HF dummy-model trick.
• We don't use the 2D-only fused_mul_mat_gguf kernel path.

Our path is a clean dequant + GEMM on N-D tensors with per-model adapters — i.e., exactly the shape of GGUF integration the upstream RFC would welcome in an OOT plugin, if we were allowed one.

On the BnB side, vllm-omni has effectively zero runtime dependency — only stale references in benchmark argparse choices and a "not tested" row in docs. That cleanup is cheap.

Dependency surface we actually lose

From vllm_omni/quantization/gguf_config.py:

from vllm import _custom_ops as ops   # -> ops.ggml_dequantize (csrc/quantization/gguf/)
from vllm.model_executor.layers.quantization.gguf import (
    UNQUANTIZED_TYPES,
    GGUFConfig,
    GGUFLinearMethod,
    LinearBase,
    QuantizeMethodBase,
    UnquantizedLinearMethod,
    is_layer_skipped_gguf,
)

Plus tests/diffusion/quantization/test_gguf_config.py imports UNQUANTIZED_TYPES from the same module.

Everything else — quantization/factory.py::_build_gguf, diffusers_loader.py routing, the four gguf_adapters/*.py files, the adapter tests — is already ours and continues to work as long as the kernel + these seven Python symbols keep resolving somewhere.

Total upstream-owned surface we depend on: one CUDA op + seven Python symbols from one file.

Proposed Change.

Four options, not mutually exclusive:

A. Vendor the minimum GGUF surface into vllm_omni/quantization/. Copy GGUFConfig, GGUFLinearMethod, UNQUANTIZED_TYPES, is_layer_skipped_gguf into vllm-omni (≤200 lines; we already subclass two of them). Replace ops.ggml_dequantize with either:

• A1 — a vendored CUDA kernel shipped under vllm-omni's csrc/, or
• A2 — a pure-Python dequant fallback via gguf.quants.dequantize() from the upstream gguf pip package. Slower for Q4_K_M / Q5_K_M, but acceptable for diffusion where matmul dominates and dequant is one-shot weight-only. Zero CUDA maintenance burden.

B. Out-of-tree quantization plugin. Register gguf back into vllm.model_executor.layers.quantization.QUANTIZATION_METHODS from vllm_omni at import time. @mgoin notes in the upstream RFC that OOT plugins for LLM GGUF are hard because of the linear.py branches. Those branches don't exist on our path, so a plugin registration could actually work for us where it doesn't for LLMs.

C. Upstream advocacy: keep the kernel + quant method module, drop only the LLM glue. Comment on vllm#39583 asking maintainers to preserve vllm._custom_ops.ggml_dequantize and vllm.model_executor.layers.quantization.gguf (the ~691-line gguf.py) even while removing model_loader/gguf_loader.py, transformers_utils/gguf_utils.py, the linear.py branches, the MoE branches, and the model-specific RoPE detection in 8+ model files. The cleanup the upstream RFC actually wants is in linear.py + fused_moe/layer.py + the loader — not the kernel itself.

D. Drop GGUF from vllm-omni. Not acceptable — it silently deletes a shipped feature and pushes consumer-GPU diffusion users off the project.

Recommendation

Pursue C + A in parallel:

1. Immediately post a comment on vllm#39583 from the vllm-omni side making the DiT case and asking for the kernel + gguf.py to be carved out (C).
2. In parallel, land a vendored fallback (A) within the feedback window so we are not blocked on the upstream decision. Even if C succeeds, decoupling narrows our contract with upstream to one explicit surface and unblocks NPU/Ascend parity work already constrained by the shared path.

BnB cleanup (separate, low-effort):

• Drop bitsandbytes from --quantization argparse choices in benchmarks/diffusion/quantization_quality.py.
• Remove the "not tested" row from docs/user_guide/diffusion/quantization/overview.md.
• Remove bitsandbytes from the registry round-trip list in tests/diffusion/quantization/test_fp8_config.py.
• No runtime code changes required.

Task breakdown

• Post a vllm-omni comment on vllm-project/vllm#39583: summarize the DiT use case, confirm we don't touch the hot paths they're cleaning up, and request that gguf.py + ggml_dequantize remain available (or be moved to a stable path).
• Vendor GGUFConfig / GGUFLinearMethod / UNQUANTIZED_TYPES / is_layer_skipped_gguf into vllm_omni/quantization/gguf_base.py. Have gguf_config.py subclass the vendored classes behind a VLLM_OMNI_USE_VENDORED_GGUF flag that defaults to the upstream import while it still exists.
• Add a pure-Python ggml_dequantize fallback via gguf.quants.dequantize guarded by hasattr(ops, "ggml_dequantize"). Benchmark Q4_K_M / Q5_K_M / Q8_0 on Flux2-Klein 4B and Qwen-Image to quantify the gap vs the CUDA op.
• Mirror the same decoupling in tests/diffusion/quantization/test_gguf_config.py and the three adapter tests.
• BnB cleanup: remove from benchmark argparse + docs overview + fp8_config test registry list.
• Update docs/user_guide/diffusion/quantization/gguf.md section 5 to reflect whichever code path we land on.

Out of scope

• Changes to gguf_adapters/* — these don't depend on upstream symbols.
• LLM-side GGUF support in vLLM (Qwen3-Omni thinker path). We don't ship that.
• New quant formats — tracked separately in #1854 and #2438.

Feedback Period.

Two weeks, aligned with the upstream vllm#39583 feedback window.

CC List.

@david6666666 @Isotr0py @hsliuustc0106 @DarkLight1337 @mgoin

Any Other Things.

If upstream agrees to keep the kernel + quant method module under a stable path (option C), most of option A collapses to a thin compatibility shim rather than a full vendored copy. Prefer that outcome.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.