[New Model]: MiniMind-O (0.1B / 0.3B-A0.1B end-to-end Omni with Thinker-Talker + Mimi codec)

Model Overview

MiniMind-O by @jingyaogong is a from-scratch, end-to-end Omni model family. A single set of weights handles text + audio + vision input and emits text + streaming audio output. Initial release on 2026-05-05.

Model	Params	License	Capability
minimind-3o	~0.1B (115M)	Apache-2.0	Dense Thinker-Talker Omni
minimind-3o-moe	~0.3B-A0.1B (312M-A115M)	Apache-2.0	MoE Thinker-Talker Omni

Architecture

• Thinker-Talker dual path, the same shape as Qwen3-Omni's Talker pipeline already supported in vllm-omni.
• Audio output: Talker emits 8-layer Mimi codec tokens (12.5 Hz, 24 kHz) through an MTP (Multi-Token Prediction) head, decoded by Mimi to a 24 kHz waveform.
• Audio input: frozen SenseVoice-Small encoder, 2-layer MLP projector.
• Vision input: frozen SigLIP2 encoder, 2-layer MLP projector.
• Streaming + barge-in: VAD-driven barge-in for near-duplex interaction; phone-mode WebUI included.
• Voice cloning: 5 built-in voices, 7 unseen-voice prompts, plus arbitrary reference audio.

The stage graph maps almost 1:1 onto the existing Qwen3-Omni serving path (Thinker → Talker → Code2Wav). The two pieces that need new integration code are the Mimi codec wrapper and the MTP audio head; everything else should reuse existing scaffolding.

Author engagement

@jingyaogong (model author) confirmed in jingyaogong/minimind-o#2 that they:

1. Welcome the integration into vLLM-Omni.
2. Will review the architecture path on any PR we land.
3. Will study the existing #2319 / #2462 / #3388 patterns and may submit a parallel PR on their side.

Worth noting from the same thread: a large portion of MiniMind-O's synthetic training data was sampled using vLLM-Omni driving Qwen-TTS, so this is also a real production user of vllm-omni proposing reciprocal integration.

Motivation

1. Architecture fit: closest open-source Omni model to our existing Qwen3-Omni stage graph; integration cost looks low.
2. Reference Omni at sub-1B scale: minimind-3o (115M) is small enough to run in CI and use as a system-level testbed for scheduler / Fast AR fusion / batch-size experiments.
3. Capability coverage: streaming, barge-in, near-duplex, voice cloning, phone mode, and vision input in one model.
4. License: Apache-2.0 across weights, data, and training code.
5. Active upstream: tech report (arXiv:2605.03937) and full training code published; author engaged and responsive.

Implementation notes

• Existing Qwen3-Omni stage graph (Thinker → Talker → Code2Wav) is the natural starting point.
• New components needed:
  • Mimi codec wrapper (8-layer codebook, 12.5 Hz, 24 kHz output) for the Code2Wav-equivalent stage.
  • MTP audio head support inside the Talker stage.
  • Frozen SenseVoice-Small + SigLIP2 projector loading on the input side.
• Stage configs go under vllm_omni/model_executor/stage_configs/.
• minimind-3o and minimind-3o-moe share the same architecture, so MoE support is the only delta between the two.

References

• GitHub: https://github.com/jingyaogong/minimind-o
• HuggingFace collection: https://huggingface.co/collections/jingyaogong/minimind-o
• Tech report: http://arxiv.org/abs/2605.03937
• Online demo: https://modelscope.cn/studios/gongjy/MiniMind-O
• Author engagement thread: https://github.com/jingyaogong/minimind-o/issues/2

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.