NVIDIA has released Nemotron 3.5 Lightning — an open model from the MoE family with 30 billion parameters, of which 3 billion are active per token, and the NeMo Switchyard proxy library for intelligent routing of requests between models. The model is available on Hugging Face under the OpenMDW-1.1 commercial license and runs on a single GPU.

image
image
image

What happened

The Nemotron 3.5 Lightning model is built on a hybrid architecture: it combines Mamba-2 for efficient processing of long contexts and Transformer layers with Multi-Token Prediction for simultaneous prediction of multiple tokens. The context window reaches 1 million tokens. The model was trained on more than 20 trillion tokens with a data cutoff of September 2025 for pretraining and May 2026 for post-training. It can run on a single GPU — from DGX Spark and H100 to RTX 5090. It comes with NeMo Switchyard — a Rust-based proxy library for routing requests between models: complex tasks are directed to powerful models, routine ones remain on lightweight ones. Speculative decoding via DSpark and DFlash accelerates inference by 60–85%. Benchmark results: 86% accuracy on PinchBench, 52.8% on SWE-bench Verified.

Context

NVIDIA is promoting an open model strategy — CEO Jensen Huang has publicly stated support for open-weight AI to accelerate the industry and stimulate GPU sales. The model is licensed under OpenMDW-1.1, commercial use is permitted. The concept of a "system of models" assumes that instead of one monolithic model for all tasks, a router is used to distribute requests between specialized models. NeMo Switchyard is integrated with LangChain, Kong AI Gateway, LiteLLM, and Nous Research platforms (Hermes). For fine-tuning, a dataset for RL agents is provided — Nemotron-RL Agentic Terminal Pivot, as well as full documentation for NeMo.

Why this matters for the industry

The "system of models" architecture changes the economics of agentic workloads. NVIDIA partners demonstrate specific results: LangChain recorded a 74% reduction in costs with a 6% loss in accuracy, Cognition (Devin) — a 28% reduction in cost while maintaining frontier-level accuracy, Boomi — 59% of traffic routed to a model 5 times faster, Classmethod — 27% savings. The MoE configuration of 30B / 3B active parameters radically reduces inference costs, making agentic workloads economically viable for a wide range of developers. A mass transition to the system of models architecture is expected: instead of one heavy model — a router plus several specialized ones. NeMo Switchyard can become the de facto standard for routing, similar to how LiteLLM became the standard for the proxy layer.

Why this matters for users

The model is available right now — weights on Hugging Face, NVIDIA NIM Playground allows testing without registration. Support in major inference engines: vLLM, Ollama, LM Studio, llama.cpp — run locally on RTX 5090 or DGX Spark. Full documentation for fine-tuning via NeMo. For products using agentic workflows via LangChain or LiteLLM, NeMo Switchyard can be integrated immediately. Direct effect: reducing inference costs at routine stages of the pipeline and freeing up resources for complex tasks.

What is still unknown / limitations

The phrase "frontier accuracy" is an exaggeration. 52.8% on SWE-bench Verified is a good result for the class of 3B active parameters, but it is far from frontier models (Claude, GPT-4 class): they achieve 70–80%+ on the same benchmarks. The long-term stability of the Mamba-2 + Transformer architecture in production workloads still needs to be confirmed. The actual 74% savings according to LangChain may vary depending on the specific workload and routing strategy.

Sources

Author

Look at AI, editorial team