💻 Baseten open-sources UltraFast engine for DeepSeek-V4.1-Flash
The Rust and CUDA inference engine is optimized for NVIDIA B200/B300 GPUs. The README lists: 2060 tokens/s decoding on 8x B300 (TP8), 1M token prefill in 11.7 seconds, 1M context, and KV-cache sharing via Redis and RDMA.
🌍 2000+ tokens/s on standard B200/B300 — Cerebras and Groq levels, but on standard GPUs. The open-source code reduces provider and self-hosting team dependency on closed runtimes, while cost and latency for long agentic sessions cease to be bottlenecks. Judging by the metrics, the speed is achieved through speculative decoding.
👤 Benchmarks are reproducible via the README; it also includes a demo script streaming Tetris (~1700–2100 tokens/s). The DeepSeek-V4.1-Flash-Fast model is available via Baseten's OpenAI-compatible API.
Source 1: https://github.com/basetenlabs/deepseek-ultrafast Source 2: https://www.baseten.co/resources/changelog/deepseek-v41-flash-fast-available-on-baseten/
