Baseten has released deepseek-ultrafast, an open-source inference engine—responsible for real-time language model response generation—written in Rust and CUDA specifically for DeepSeek-V4.1-Flash on NVIDIA B200 and B300 server GPUs. According to the README, the engine delivers approximately 2000 tokens per second during decoding and processes a prefill of one million input context tokens in 11.7 seconds. The DeepSeek-V4.1-Flash-Fast model is already available via Baseten's OpenAI-compatible API, and the open code allows teams with suitable hardware to independently reproduce the claimed benchmarks.

image
image
image

What happened

Baseten published the basetenlabs/deepseek-ultrafast repository containing a Rust and CUDA engine written for DeepSeek-V4.1-Flash, targeting NVIDIA B200 and B300 GPUs with sm_100a and sm_103a architectures. According to the README, decoding reaches 2060 tokens per second on a setup of eight B300s in a TP8 configuration: one step takes 2.71 milliseconds with an average accepted block length of 5.6 tokens, while 1740 tokens per second are claimed on four GB300 accelerators. Prefilling one million tokens fits within 11.7 seconds, equivalent to 85,000 tokens per second. The engine supports a 1-million-token context and can share the KV-cache across multiple servers via Redis and RDMA. According to the authors, GPU kernels were refined by coding agents working with FlashInfer and SGLang sources, and the DeepSeek-V4.1-Flash-Fast model is delivered to clients via Baseten Model APIs through the OpenAI-compatible endpoint inference.baseten.co.

Context

Speeds comparable to specialized inference chips from Cerebras and Groq were previously only available on their proprietary hardware: these companies build their business on instant single-stream generation. The ratio of a 2.71-millisecond step to an average accepted block length of 5.6 tokens in the README arithmetically indicates speculative decoding—a technique where draft tokens are proposed in advance and verified by the main model in blocks, meaning the final speed depends on the proportion of accepted proposals. The author of the original Telegram post recalls their joke that engineers should write their own scaffolds, calling this development a new milestone: coding agents with a clear target metric in tokens per second are refining GPU kernels themselves, relying on FlashInfer and SGLang sources. The author interprets this practice as an early example of recursive self-improvement (RSI), although humans are still involved in the optimization loop.

Why this matters for the industry

The main industry impact concerns the economics of inference: if the figures are confirmed by independent measurements, single-generation latency levels comparable to Cerebras and Groq will become achievable on standard server GPUs for a model with open weights, and specialized accelerators will lose their exclusivity on instant responses. The open engine allows inference providers and teams deploying models in-house to avoid reliance on closed proprietary stacks and to modify the code for their own pipelines. For agentic products, another aspect is more important: sessions with contexts up to one million tokens no longer hit decoding speed limits, and prefill at 85,000 tokens per second reduces the latency cost of the "read a large corpus first, then answer" scenario, shifting the bottleneck to prefill and orchestration. Likely consequences include engine forks, attempts to port the approach to other popular models, and intensified provider competition on speed at batch=1, so product teams should consider designing interfaces with thousands of tokens per second in mind.

Why this matters for users

You can read and verify this immediately: the repository contains a README with instructions and a demo script that streams a Tetris game at a speed of approximately 1700–2100 tokens per second, and teams with B200, B300, or GB300 GPUs have everything needed to reproduce the benchmarks on their own hardware. For those without such accelerators, it is sufficient to connect DeepSeek-V4.1-Flash-Fast via the OpenAI-compatible endpoint inference.baseten.co and measure the actual speed in tokens per second, time to first token, and answer quality on their own tasks within a day. The practical significance for readers: interactive assistants and agents at this speed respond almost instantly, so long sessions with huge contexts no longer feel slow. However, hardware requirements are extreme, so for most teams this is currently a benchmark for a new upper speed limit and a reason to reconsider pricing expectations, rather than an immediate replacement for their working stack.

What is currently unknown / limitations

All key figures are taken from Baseten's own README and refer to the batch=1 mode after warm-up: there are no independent public replications or detailed measurement methodologies yet, so the metrics should be treated as a hypothesis subject to verification. Due to speculative decoding, the actual speed will depend on the proportion of accepted draft tokens for a specific model and load. Direct comparison with Cerebras and Groq is incorrect: this is a measurement of single-stream latency for a single request versus metrics from specialized accelerators obtained on other models and methodologies. The engine only runs on sm_100a and sm_103a architectures, i.e., on B200, B300, and GB300, and will not work on other hardware. Finally, the thesis about recursive self-improvement remains the post author's interpretation, not an established fact: humans are still participating in the optimization loop.

Sources

Author

Look at AI, editorial team