Chinese company StepFun has released the StepAudio 3 family of audio models. Its flagship, StepAudio 3 Realtime, is a full-duplex audio-language model that reasons in parallel with speech: it asks clarifying questions, maintains turns, and can be interrupted without pauses. The lineup also includes StepAudio 3 Music for generating full songs from text descriptions and StepAudio 3 Gen for unified design of voice, effects, ambience, and music, while ASR Max and TTS cover recognition and speech synthesis — the entire audio pipeline is assembled under a single API key.

image
image

What Happened

StepFun announced the StepAudio 3 family across several directions. The flagship StepAudio 3 Realtime is built on a "listen — speak — think — act" architecture: in Think-While-Speaking mode, internal reasoning runs in parallel with speech, so the model can ask clarifying questions, maintain turns, and be interrupted without pauses; working languages are Chinese and English. StepAudio 3 Music generates full songs with and without lyrics: first, chain-of-thought planning ABC-COT sets the track structure — verses, chorus, bridge — and arrangement, then an AR + DiT combination synthesizes a finished 48 kHz stereo recording up to 5 minutes 30 seconds long; claimed modes include song generation from description, instrumental, "vocal into song," and cover. StepAudio 3 Gen was also announced — unified generation of voice with voice design, sound effects, ambience, and music; its description is in technical report arXiv:2609.12945, and the tier is called stepaudio-3-gen-preview. A technical report for Realtime was published at arXiv:2609.14005, and official demo pages for Realtime and Music are already open.

Context

To assess the announcement, it is useful to understand how full-duplex differs from familiar turn-based assistants: a classic voice bot waits for the end of a turn, while Realtime combines listening, speaking, and reasoning in a single stream, and according to the claimed architecture can perform agentic actions directly in dialogue — built-in VAD, ASR, web_search, and retrieval are available in the Realtime API via WebSocket. The vendor's numbers look strong: 98.9 on Artificial Analysis Full-Duplex Bench versus 95.3 for GPT-realtime-2 (High) and 98.4 for Qwen Audio 3.0 Realtime Plus, as well as 90.6 on MMSU. However, 56.0 on τ-Voice is an honest indicator of maturity: "reasoning inside voice" is still far from text-level reasoning; models are good at chatting, but harder to think in real time. In recognition, 0.57% error is claimed in a contextual test and minus 43% compared to Doubao ASR 2.0; the 1.18% result on LibriSpeech clean is uninformative because it is a nearly saturated benchmark with low discriminative power. TTS supports streaming output until the end of sentence synthesis, meaning speech starts sounding before the sentence is fully synthesized.

Why This Matters for the Industry

StepFun covers the entire audio pipeline with one lineup — recognition, speech synthesis, audio design, and full-duplex dialogue — and sells it at prices below incumbents: ASR $0.24 per hour, TTS $0.36 per 10,000 characters, Gen free during the trial. For startups, this is a sharp reduction in the cost of entry into voice products: full-duplex interruptions, VAD, and ASR are available "out of the box," and a voice agent prototype can be assembled without separate code for speech annotation. The flip side is the devaluation of simple wrappers around ASR and TTS: value shifts from model access to orchestration, distribution, and vertical scenarios. Pressure will be felt by OpenAI Realtime API, Gemini, and Doubao — in price and in scenarios where interruptions and agentic actions without dialogue breaks are important. If independent runs confirm the Realtime numbers, full-duplex and built-in VAD/ASR risk becoming basic requirements for voice APIs, and then vendor comparison will shift to price per hour of dialogue and latency.

Why This Matters for Users

The models can be tried now: in Voice Studio at audio.stepfun.ai, conversational demos are working, and an API key opens stepaudio-3-realtime-preview, stepaudio-3-tts with voice design via text description, and the free-for-trial stepaudio-3-gen-preview. Those making voice bots, podcasts, or video narration save code on VAD and ASR — these functions are included in the Realtime API, prices for ASR and TTS are known in advance, so the pipeline can be budgeted without surprises. StepAudio 3 Music, from a description like "verse — chorus — bridge," promises a finished 48 kHz stereo track up to 5:30, which can be listened to and downloaded immediately — this is potentially ready-made material for jingles, podcast packaging, and video narration.

What Is Still Unknown / Limitations

Most of the numbers are still vendor-provided: technical report arXiv:2609.14005 gives a verifiable starting point, but there are no reproduced independent results. Realtime has only two languages — Chinese and English, and latencies are not published, so they will have to be measured independently. The methodology for comparing ASR with Doubao ASR 2.0 is not disclosed in the presentation. Music generation in Voice Studio is marked as "Coming Soon," so the claimed 48 kHz and duration up to 5:30 cannot be verified live, and it is unknown whether the ABC-COT approach will hold up on long forms. There is no data on open weights in the sources. Finally, "the entire pipeline under one API key" is a product integration of components, not scientific novelty, and the technical level of the models cannot be inferred from prices.

Sources

Author

Look at AI, editorial team