[New Model]: VoXtream2 (herimor/voxtream2)

The model to consider.

https://huggingface.co/herimor/voxtream2

VoXtream2 — a 0.5B parameter zero-shot full-stream Text-to-Speech model with dynamic speaking-rate control.

• Architecture: Fully autoregressive, codec-based TTS with three transformer components:
  • Incremental Phoneme Transformer (streaming phoneme processing)
  • Temporal Transformer (semantic + duration tokens via monotonic alignment)
  • Depth Transformer (acoustic codec token generation)
• Audio codec: Mimi (Kyutai), 16 codebooks
• Speaker encoder: ReDimNet for zero-shot voice cloning from 3-10s audio prompt
• Output: 24kHz streaming WAV audio
• VRAM: 2.2-4.2 GB
• Latency: 74ms first-packet, 4x faster than real-time on RTX 3090
• Key features: Dynamic mid-utterance speaking-rate control via distribution matching + classifier-free guidance, textless voice prompting (prompt-text masking)
• Paper: arXiv:2603.13518 (Interspeech 2026 submission)
• License: CC-BY-4.0 (weights), MIT (code)

The closest model vllm-omni already supports.

Qwen3-TTS — both are autoregressive codec-based streaming TTS models. Key differences:

• VoXtream2 uses Mimi codec (16 codebooks) vs Qwen3-TTS's custom codec
• VoXtream2 has a 3-stage transformer pipeline (phoneme → temporal → depth) vs Qwen3-TTS's talker + code predictor
• VoXtream2 is significantly smaller (0.5B vs 1.7B) with lower VRAM requirements
• VoXtream2 supports dynamic speaking-rate control mid-utterance

What's your difficulty of supporting the model you want?

• Multi-stage pipeline: The 3-transformer architecture (phoneme → temporal → depth) maps naturally to vllm-omni's multi-stage pipeline, similar to how Qwen3-TTS uses Stage 0 (Talker) and Stage 1 (Code2Wav).
• Custom codec: Uses Mimi codec with 16 codebooks — would need a decoder stage similar to Qwen3-TTS's Code2Wav stage.
• Speaker encoder: ReDimNet speaker encoder for zero-shot cloning needs to run as a preprocessing step on the reference audio.
• Phonemizer dependency: Requires eSpeak NG for phoneme conversion as a preprocessing step.
• Streaming: The model is designed for full-stream inference (74ms TTFP), which aligns well with vllm-omni's streaming audio API (/v1/audio/speech with stream=True).
• CFG (classifier-free guidance): Uses CFG across conditioning signals — may need special handling during batched inference.

Use case and motivation

• Ultra-low latency TTS: 74ms first-packet latency makes it ideal for real-time conversational AI and voice assistants.

• Extremely lightweight: Only 2.2-4.2 GB VRAM — can run on consumer GPUs and be co-located with LLMs on the same device.

• Zero-shot voice cloning: Clone any voice from a short (3-10s) audio prompt without fine-tuning, with no transcript needed for the prompt audio.

• Dynamic speaking rate: Unique ability to adjust speaking speed mid-utterance — useful for expressive speech synthesis and accessibility.

• Open weights + open code: CC-BY-4.0 weights + MIT code, fully reproducible with published training data.

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.