DeepSeek has open-sourced DeepInfer, the inference engine that powers its production services and enables the operation of the DeepSeek-V4.1-Flash model. The repository at github.com/deepseek-ai/DeepInfer was published on October 20, 2026, under the Apache 2.0 license and supports NVIDIA SM100 (Blackwell) and Huawei Ascend 950 accelerators. For the first time, the lab has released not just model weights, but a complete working production stack: according to the company, the engine delivers up to 155 trillion tokens per day on a cluster, with a time to first token (TTFT) of approximately 400 ms and a generation speed of 230 tokens per second, although these figures have not yet been verified by independent benchmarks.

image
image

What happened

On October 20, 2026, DeepSeek published the repository at github.com/deepseek-ai/DeepInfer under the Apache 2.0 license. The release is not a research prototype but an inference framework operating in the company's production services and powering DeepSeek-V4.1-Flash; by default, the engine loads the DeepSeek-V4.1-Flash-1017 checkpoint from Hugging Face. The package includes optimized kernels (FlashMLA, DeepGEMM, DeepEP, and TileKernels), an Attention DP and MoE EP parallelism scheme with the EPLB balancer, DSpark speculative decoding, an asynchronous scheduler, and deterministic sampling. It supports NVIDIA SM100 (Blackwell) and Huawei Ascend 950 accelerators. According to DeepSeek's own data, the framework delivers up to 155 trillion tokens per day on a cluster, with a TTFT of approximately 400 ms and a generation speed of 230 tokens per second—up to 300 for code. However, the release does not include KV-cache, prefill/decode disaggregation, or load balancing, and the engine operates in offline inference mode.

Context

Inference is the stage where a trained model actually responds to users, and it accounts for the majority of operational costs for large models. Previously, labs primarily published weights while keeping serving infrastructure closed: third-party teams built their own stacks on universal engines like vLLM and SGLang and optimized them themselves for specific architectures. DeepInfer stands out because it is code that actually serves a commercial service, meaning performance claims are for the first time moving from a presentational plane to a potentially verifiable one: the same stack and weights are available to any team. Individual components were previously known to the community separately, so the scientific novelty of the release is moderate: the value lies in the system integration for DeepSeek architectures with a mix of experts and MLA-style attention, and in honestly documented limitations—this is an engineering, not a research, step.

Why this matters for the industry

For the industry, this is the first time DeepSeek has open-sourced an entire production inference stack, rather than just the model. It provides an independent alternative to vLLM and SGLang, pre-optimized for DeepSeek architectures: deep system integration instead of a universal framework requiring manual tuning. For companies building products on DeepSeek models, this is a direct path to reducing the cost and energy consumption of large-scale inference, and for the market, it is a blow to the moats of providers whose advantage lay precisely in efficient model serving: the barrier to entry for teams independently hosting large MoE models drops significantly. Support for Huawei Ascend 950 in the official release alongside NVIDIA SM100 is a notable step for the Chinese chip ecosystem and a strong signal of the stack's portability beyond a single vendor platform. Within six months, the first independent benchmarks of DeepInfer against vLLM and SGLang are likely: if the figures are confirmed at least partially, the engine could become a baseline for comparing DeepSeek models and put pressure on open serving frameworks to favor deep optimization for specific architectures. If the trend of open-sourcing production stacks solidifies over the next two years, the margin for pure API pass-throughs may compress, and value may shift to memory and RAG layers, orchestration, user interfaces, and data; consolidation around specialized engines and a growing role for non-NVIDIA accelerators are possible.

Why this matters for users

Developers can clone the repository and run DeepSeek-V4.1-Flash independently right now: this requires Linux, Python 3.12, and either CUDA 13 for NVIDIA SM100 (Blackwell) or CANN 9.2 for Huawei Ascend 950, with the DeepSeek-V4.1-Flash-1017 checkpoint downloading by default from Hugging Face. A practical plan for first steps: define your own load profile—typical context lengths and request parallelism levels—then run a benchmark against vLLM and SGLang and compare your TTFT and generation speed measurements with DeepSeek's figures. DSpark speculative decoding, EPLB balancing, and deterministic sampling (batch-invariant inference) deserve separate study: reproducible model behavior independent of batch composition simplifies debugging, version comparison, and precise A/B experiments. For those without suitable hardware, the release is still useful as a reference for engineering techniques: the optimized kernels, asynchronous scheduler, and parallelism schemes can be studied as a textbook on high-performance inference.

What is still unknown / limitations

The performance metrics claimed by DeepSeek have not yet been independently reproduced: verifying them requires clusters of NVIDIA SM100 or Huawei Ascend 950, which most teams do not have. Performance on Huawei Ascend 950 with CANN 9.2 cannot be taken at face value—independent measurements under real loads are required, rather than trust in vendor figures. The functional scope of the release is already documented: only SM100 and Ascend 950 are supported, it works only in offline inference mode, and KV-cache, prefill/decode disaggregation, and load balancing are not included, so the engine is not currently ready for online services with high user parallelism. Finally, the conclusion that inference is becoming a commodity does not technically follow from the fact of the repository's publication: this is a business hypothesis that will require confirmation by independent benchmarks and long-term project support.

Sources

Author

Look at AI, editorial team