Xiaomi has published MiDashengLM-Gen — an end-to-end model that, from structured text in a single stream, generates speech, music, sound effects, and acoustic environment, replacing a pipeline of separate audio models. The checkpoint, code, and online demo are already available to everyone, and the model can be run locally.

image
image
image

What happened

The team (repository xiaomi-research, model published under the name mispeech) released MiDashengLM-Gen and simultaneously published the paper MiDashengLM-Gen: Unified Audio Scene Generation via LLM-Driven Autoregressive Flow Matching (arXiv:2608.11804), submitted on August 12, 2026. The system consists of a fine-tuned Qwen3-1.7B language model, the DashengTokenizer audio tokenizer (768-dimensional latents at 25 Hz, compressed by a projector to 5 Hz), and a DiT for flow matching (16 layers, hidden 2048, 8 heads). Inference runs with a 10-step Euler ODE solver with CFG 2.0, outputting a mono WAV at 16 kHz, 1 to 20 seconds long. The scene description is written in structured text with tags , , , , , , which explicitly indicate which sound component each fragment generates. Training was conducted on the private ACAVCaps superset of 77,000 hours (speech, music, effects, derived from ACAV100M) and on TTS corpora Emilia, LibriTTS, LJSpeech, AISHELL-3, WenetSpeech4TTS.

Context

The novelty of the model is not in the generative block: the combination of LLM with flow matching on DiT is a known recipe. What is new is the unification of speech, music, sound effects, and acoustic environment synthesis in a single model with separate control of components via tags. The authors directly state that the classical approach is built on a pipeline of separate TTS, music, and SFX models with a "frozen" text encoder, which limits cross-modal optimization and speech intelligibility. MiDashengLM-Gen continues Xiaomi's line: its predecessor is the Dasheng AudioGen model, and according to the authors, MiDashengLM-Gen became the first open end-to-end generator of mixed audio scenes.

Why this matters for the industry

The component stack "TTS + music + SFX" as a standalone product is being devalued: the cost of generating a mixed audio scene drops to the level of a single open checkpoint under Apache-2.0, and teams can assemble a "text to audio track" prototype in days without external vendors. This intensifies competition in creator-tools and interactive media prototyping. The numbers show the scale of the step: on the Seed-TTS benchmark, English WER dropped to 2.79% compared to 12.15% for the predecessor Dasheng AudioGen — an improvement of about 4 times, and the quality of music and effects is competitive (FAD/CLAP on MusicCaps and MECAT). At the same time, the specialized Qwen3-TTS shows a WER of 1.24%, meaning that in pure speech intelligibility, dedicated TTS is still twice ahead, and the "LLM backbone + flow matching" architecture has not yet overtaken separate synthesizers.

Why this matters for users

The model can be tried immediately: the online demo xingws.github.io/midashenglm-gen-demo contains examples in 7 categories — SFX, music, mixed sound, 9 languages, emotions, complex phonetics. The checkpoint mispeech/midashenglm-gen is on Hugging Face with a three-line quickstart (AutoModel.from_pretrained + generate), and the repository has infer.py for local launch. Already today, it is realistic to build an MVP of "audio track from text" for short fragments up to 20 seconds: jingles, sound effects, audio inserts for advertising, game scene prototypes.

What is still unknown / limitations

Output is mono 16 kHz, below CD quality; scene length is limited to about 20 seconds, larger clips have not been tested by the authors. There is no voice cloning in the model. For the Russian language, the model is not yet ready for production: WER 13.19% compared to 2.42% in English, and the reason is the composition of the data: Russian is not in any of the TTS corpora on which the system was trained, and in the speech slice of ACAVCaps its share is only 4.66% compared to 54% for English and 24.5% for Chinese. Data on latency, API, and pricing has not been published, production-readiness remains unverified.

Sources

Author

Look at AI, editorial team