[New Model]: stabilityai/stable-audio-3-medium (DiT text-to-audio for music and SFX)

The model to consider.

Stable Audio 3, the next generation of Stable Audio from Stability AI:

• Medium (GPU/CUDA, primary serving target): https://huggingface.co/stabilityai/stable-audio-3-medium — 1.4B params, up to 380s
• Small-Music / Small-SFX (CPU, 433M each, up to 120s): https://huggingface.co/stabilityai/stable-audio-3-small-music, https://huggingface.co/stabilityai/stable-audio-3-small-sfx
• Full collection: https://huggingface.co/collections/stabilityai/stable-audio-3
• Reference implementation (MIT): https://github.com/Stability-AI/stable-audio-3
• Technical report: https://arxiv.org/abs/2605.17991

Stable Audio 3 Large (2.7B) is API-only and out of scope.

The closest model vllm-omni already supports.

stabilityai/stable-audio-open-1.0 (#2917). Stable Audio 3 is its direct successor: same diffusion-transformer family and the same text-to-audio task. The pipeline shape (text encoder -> DiT denoiser -> audio autoencoder decode) carries over, so the existing Stable Audio Open integration is the natural starting point.

What's your difficulty of supporting the model you want?

A few things differ from stable-audio-open-1.0:

• New autoencoder. Stable Audio 3 ships SAME (Semantic-Acoustic Music Encoder): stereo, 44.1 kHz, 256-dim latents, with separate Small/Large variants. The decoder is new code and supports chunked decoding to cap VRAM; it is not the stable-audio-open VAE.
• No diffusers pipeline. Unlike stable-audio-open-1.0 (which has StableAudioPipeline in diffusers), Stable Audio 3 only ships the standalone stable_audio_3 package in the Stability repo. The DiT and autoencoder adapters must be ported from that repo rather than reused from diffusers.
• Variable-length generation. The model sizes latents to the requested duration instead of a fixed window, which affects how the scheduler batches requests.
• Three inference modes. Beyond text-to-audio it supports audio-to-audio editing and inpainting/continuation, both of which take a reference-audio input. Initial support can scope to text-to-audio only.
• Misc. Medium requires Flash Attention 2; LoRA adapters are stackable and runtime-adjustable, which may want a serving-side knob later.

Use case and motivation

Stable Audio 3 is the current open-weight SOTA for text-to-audio music and sound-effect generation, and is a straight upgrade over stable-audio-open-1.0 in quality, max length (up to 380s for Medium), and speed. It fits vllm-omni's existing diffusion text-to-audio path, and supporting it keeps audio-generation coverage current. The Medium checkpoint is modest (1.4B, ~6.5 GB peak VRAM), so it serves on a single commodity GPU.