Major infrastructure provider Hetzner has begun testing its own API for large language model inference. The solution is compatible with the OpenAI protocol and offers high performance based on the Qwen model family.
What Happened
Hetzner introduced an experimental API supporting the Qwen/Qwen2.5-32B-Instruct-FP8 model. This is a Mixture-of-Experts (MoE) architecture, where only 3 billion out of 35 billion total parameters are active at any given time. Tests demonstrate a Time To First Token (TTFT) of 153 ms and a throughput of up to 224 tokens per second, with support for a 262K token context window.
Context
Hetzner's entry into this market marks a shift from specialized AI startups to large, traditional hosting providers. The use of MoE architecture allows for efficient distribution of computational resources, providing speeds comparable to solutions designed for low-latency tasks.
Why It Matters for the Industry
The arrival of players like Hetzner could turn LLM inference into a low-margin infrastructure commodity. This will intensify competition among cloud providers, potentially lowering computation costs and forcing specialized service providers to find new ways to protect their profit margins.
Why It Matters for Users
Developers gain the ability to use modern open-weights models through the familiar OpenAI interface on alternative infrastructure. This can be significantly more cost-effective than using proprietary APIs and simplifies model deployment with minimal changes to existing code.
What Is Not Yet Known / Limitations
There are potential risks regarding corporate compliance and data security when using an alternative cloud provider, which require separate evaluation for the enterprise segment.
Sources
Author
Look at AI, Editorial Team