[New Model]: LongCat-AudioDiT (Meituan) — Waveform Latent Space Diffusion TTS

The model to consider.

LongCat-AudioDiT by Meituan LongCat team — a diffusion-based zero-shot TTS model that operates directly in waveform latent space, achieving SOTA speaker similarity on the Seed benchmark.

Model	Weights	Size	Key Capability
LongCat-AudioDiT-1.1B	Public (MIT)	1.1B	Zero-shot TTS with waveform latent diffusion
LongCat-AudioDiT-3.5B	Public (MIT)	3.5B	SOTA zero-shot voice cloning (SIM 0.818 on Seed-ZH)

The closest model vllm-omni already supports.

CosyVoice 3.5 / Qwen3-TTS — both are diffusion-based TTS models already supported in vllm-omni. LongCat-AudioDiT shares the flow-matching diffusion paradigm but differs in:

1. Waveform latent space: Uses a Wav-VAE to encode raw waveforms into a 64-dim latent space at 11.7Hz (2048x compression), bypassing mel-spectrogram intermediates entirely.
2. Single-stage pipeline: Wav-VAE encoder → DiT flow matching → Wav-VAE decoder. No separate vocoder or codec stage.
3. Text encoder: UMT5-base (supports 107 languages) with a novel first-layer + last-layer embedding fusion for improved intelligibility.

What's your difficulty of supporting the model you want?

Architecture Overview

The model consists of two components:

1. Wav-VAE (fully convolutional audio autoencoder):

   • Oobleck blocks with dilated convolutions for multi-scale modeling
   • Snake-Beta activations
   • Non-parametric shortcut branches for training stability
   • 24kHz → 11.7Hz frame rate (2048x compression ratio)

2. DiT Transformer (diffusion backbone with conditional flow matching):

   • 1536 hidden dims, 24 layers, 24 attention heads
   • Global AdaLN for timestep injection
   • QK-Norm + RoPE (Qwen2-style rotary embeddings)
   • Long-skip connections
   • REPA (representation alignment via mHuBERT features)

Key Technical Challenges

• New operators: Snake-Beta activation in Wav-VAE, Adaptive Projection Guidance (APG) replacing standard CFG during inference
• Training-inference mismatch fix: Prompt region latent reset + unconditional prediction purification at each ODE step — this is a novel inference-time technique
• Flow matching inference: Euler ODE solver (default 16 steps) with configurable guidance strength
• HuggingFace integration: Already registered with transformers via AutoConfig / AutoModel, with custom AudioDiTConfig, AudioDiTVaeConfig, and AudioDiTModel

Pipeline Mapping

This is a single-stage diffusion pipeline (no autoregressive component):

• Input: text tokens (UMT5) + optional prompt audio (for voice cloning)
• Process: Flow matching in waveform latent space
• Output: Decoded waveform via Wav-VAE

Similar to how image diffusion models are served, but for audio generation.

Use case and motivation

1. SOTA zero-shot voice cloning: LongCat-AudioDiT-3.5B achieves 0.818 SIM on Seed-ZH and 0.797 on Seed-Hard, surpassing Seed-TTS, CosyVoice 3.5, and MiniMax-Speech.
2. Competitive intelligibility: CER 1.09% (ZH), WER 1.50% (EN) — top-tier among NAR models.
3. MIT license: Fully open-source and commercially usable.
4. Efficient single-stage design: No cascaded vocoder means fewer error accumulation points and simpler serving architecture.
5. Multilingual: UMT5 text encoder supports 107 languages out of the box.
6. Active community interest: The model was released alongside a research paper demonstrating significant advances in waveform-space TTS.

References

• Paper: https://arxiv.org/abs/2603.29339v1
• GitHub: https://github.com/meituan-longcat/LongCat-AudioDiT
• HuggingFace: https://huggingface.co/meituan-longcat/LongCat-AudioDiT

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.