NVIDIA has introduced Audex, an innovative unified audio-text model based on the Nemotron 3 Nano architecture. The 30B version, which utilizes a Mixture of Experts (MoE) mechanism with 3B active parameters, is capable of processing up to 1 million tokens of context and audio recordings up to 15 minutes long, combining sound understanding, speech generation, and textual reasoning into a single system.
What Happened
NVIDIA released the Audex model, which implements the Unified Audio Language Models (UALM) approach. The technical foundation includes the use of NV-Whisper for audio encoding and XCodec/XCodec-2 for tokenization. The model supports two operating modes: Thinking and Instruct, allowing it not only to process commands but also to perform complex logical reasoning based on audio data. The weights for the 30B and 2B versions are already available on Hugging Face.
Context
Traditionally, voice processing was built on cascaded systems, where separate models were connected sequentially: Automatic Speech Recognition (ASR), a Large Language Model (LLM), and Text-to-Speech (TTS). This approach often led to latency and the loss of contextual nuances when transferring data between modules. Audex aims to replace this chain with a single native multimodal architecture.
Why It Matters for the Industry
For the AI industry, this means a radical simplification of developing complex voice interfaces. The transition to UALM reduces latency and allows for the construction of systems with a deep understanding of the acoustic environment, including audio event recognition. The availability of open weights encourages researchers to implement advanced audio reasoning into their pipelines, bypassing the need to assemble cumbersome pipelines from several discrete models.
Why It Matters for Users
For end users, this paves the way for creating truly intelligent voice assistants. Unlike current systems that merely translate sound into text, Audex-level technology allows devices to "hear" intonations, emotions, and specific sound events, providing more natural and intelligent real-time interaction.
What Is Not Yet Known / Limitations
Despite the technical advantages in reducing latency, the use of such models creates new legal and ethical risks in the areas of biometrics and personal data protection, which require additional regulation.
Sources
Author
Look at AI, Editorial Team