Speculative decoding allows for increasing the speed of local language models by 1.5–2.5x without losing response quality, by using small auxiliary models to accelerate the generation process.
What Happened
Speculative decoding technology uses a small and fast model (draft model) to pre-predict tokens, which are then verified by the main large model in a single pass. Modern architectures, such as Qwen 3.6 and Gemma 4, are already implementing Multi-Token Prediction (MTP) mechanisms, where the speculation process is integrated directly into the model's structure.
Context
Traditionally, acceleration was achieved by externally pairing a main model with a draft model. However, the transition to an architecturally built-in MTP mechanism allows for optimizing GPU memory bandwidth usage, which is a critical bottleneck during large model inference.
Why It Matters for the Industry
For the industry, this shift signifies a move from using external optimizations to creating new training standards. Implementing MTP in SOTA models will make speculative decoding a standard "out-of-the-box" feature, increasing the economic efficiency of inference and reducing the Total Cost of Ownership (TCO) for large AI services.
Why It Matters for Users
Users running models locally via Ollama, LM Studio, or llama.cpp can significantly reduce latency and speed up text generation. This is especially noticeable in coding tasks and when working with interactive AI agents.
Sources
Author
Look at AI, Editorial Team