[New Model] PrismAudio (Video-to-Audio Generation) · vllm-project/vllm-omni#2140

2026-03-24T15:00:05.000Z

### Model [PrismAudio](https://prismaudio-project.github.io/) — a video-to-audio generation framework from FunAudioLLM. Generates environment sounds and sound effects synchronized with video content (e.g. footsteps, rain, metal impacts). 518M parameters, generates 9s audio in 0.63s. Published at ICLR 2026. ### Architecture Diffusion-based model with decomposed chain-of-thought reasoning. Uses four reward models (semantic, temporal, aesthetic, spatial) for multi-objective optimization via reinforcement learning (Fast-GRPO). Key components: - Diffusion backbone for audio generation - Chain-of-thought decomposition for multi-dimensional planning (semantic, temporal, aesthetic, spatial) - Multi-reward RL training with Fast-GRPO for efficient optimization ### Fit with vllm-omni Single-stage diffusion pipeline, similar to Stable Audio and AudioX. Could reuse the existing diffusion infrastructure (DiffusionAttention, pipeline base classes). ### References - Project page: https://prismaudio-project.github.io/ - Paper: arXiv:2511.18833 - HuggingFace: https://huggingface.co/FunAudioLLM/PrismAudio - ModelScope: https://www.modelscope.cn/models/iic/PrismAudio - Demo: https://huggingface.co/spaces/FunAudioLLM/PrismAudio

(4 comments) (0 reactions) (1 assignee)Python (1,067 forks)github user discovery

good first issuehelp wantednew model

Repository metrics

Stars: (4,990 stars)
PR merge metrics: (PR metrics pending)

Description

Model

PrismAudio — a video-to-audio generation framework from FunAudioLLM. Generates environment sounds and sound effects synchronized with video content (e.g. footsteps, rain, metal impacts). 518M parameters, generates 9s audio in 0.63s. Published at ICLR 2026.

Architecture

Diffusion-based model with decomposed chain-of-thought reasoning. Uses four reward models (semantic, temporal, aesthetic, spatial) for multi-objective optimization via reinforcement learning (Fast-GRPO).

Key components:

Diffusion backbone for audio generation
Chain-of-thought decomposition for multi-dimensional planning (semantic, temporal, aesthetic, spatial)
Multi-reward RL training with Fast-GRPO for efficient optimization

Fit with vllm-omni

Single-stage diffusion pipeline, similar to Stable Audio and AudioX. Could reuse the existing diffusion infrastructure (DiffusionAttention, pipeline base classes).

References

Project page: https://prismaudio-project.github.io/
Paper: arXiv:2511.18833
HuggingFace: https://huggingface.co/FunAudioLLM/PrismAudio
ModelScope: https://www.modelscope.cn/models/iic/PrismAudio
Demo: https://huggingface.co/spaces/FunAudioLLM/PrismAudio

Contributor guide

Research direction: Study the PrismAudio paper and codebase, then integrate the model into vllm omni's diffusion pipeline by reusing existing infrastructure like DiffusionAttention and pipeline base classes.
Tech stack: pythonpytorch
Domain: machine learningai
Issue type: Feature
Difficulty: 4
Estimated time: 3-5 days
Activity status: Active
Clarity: Clear
Prerequisites: PythonPyTorchDiffusion Models
Newbie friendliness: 30