Together AI has introduced Together Compiler (TKCC)—a "continual compilation" system for AI inference where coding agents constantly rewrite model forward passes to match current traffic and hardware, while the inference engine measures candidates without restarts. According to the company, engine startup time has been reduced from 467 to 9 seconds, and in less than 24 hours, the system increased generation speed by up to 2.6x on open models and set a record in the SemiAnalysis InferenceX benchmark.

image
image
image

What Happened

Together AI announced a system where agents with access to modify inference code continuously optimize the model's forward pass for changed load profiles or new equipment. The results of each optimization candidate are automatically verified through the inference engine to prevent quality degradation. In the first 24 hours of operation, the approach yielded up to 2.6x acceleration in generation on open models.

Context

Traditionally, inference engines and serving stacks are optimized manually by engineers for specific load profiles and hardware. This means that every change—increased prompt lengths, model swaps, or transitions to new GPUs (H100, B200, B300, NVL72, Groq)—requires new manual tuning, and the effect gradually "expires" as traffic drifts. Together AI proposes to convert this process into a continuous closed loop of "agent rewrites forward pass → engine instantly measures candidate": the key engineering element is a persistent inference engine, whose startup time has been reduced from 467 to 9 seconds, providing 252x more experiments per unit of time. It is the speed of measurement that makes automatic optimization practical: the system sets SOTA in the SemiAnalysis InferenceX benchmark—1111 TPS on MiniMax M3.1-Flash (B200) and 1205 TPS on DeepSeek v4.1-Flash (GB300). A notable detail: "naive" agents without expert hints showed 6.2x acceleration compared to 4.9x for expert strategies—if this is sustainable, it challenges the common assumption that "strong expert prompts are better" in agent-based system tuning.

Why This Matters for the Industry

For inference providers and teams serving models in production, continual compilation can reduce GPU costs: the system automatically adjusts planning and computations to real traffic instead of manual tuning for peak scenarios. This reduces the need for highly specialized CUDA kernel engineers: part of the work on profiling and rewriting low-level optimizations shifts to agents. Competitors will have to respond with their own agent-based optimization systems to avoid falling behind in price per token—otherwise, a 2.6x difference on identical models will become noticeable in unit economics.

Why This Matters for Users

For API inference users, acceleration manifests as lower latency and reduced price per token with unchanged model quality. For teams deploying open models (Kimi K3, GLM 5.3, MiniMax M3.1-Flash, DeepSeek v4.1-Flash) on their own hardware, questions of applicability arise: the claimed speeds were obtained on provider equipment, and independently replicating the results requires access to the same hardware configuration and agent loop.

What Is Still Unknown / Limitations

Key figures (2.6x TPS, 467→9 seconds for engine startup, 252x experiment throughput, 6.2x for naive agents vs. 4.9x for expert strategies, 1111 TPS and 1205 TPS InferenceX) are taken from Together AI's vendor blog and media reports without independent verification. The distribution of accelerations (median, minimum), measurement conditions, TP/EP/DCP configurations, and iteration budgets when comparing "naive" and "expert" agents have not been disclosed. For LLMs, output quality after rewriting the forward pass is not claimed to be bit-exact—this limitation has only been confirmed for scientific models (weather, PDE, time-series). The company has not described ways to apply TKCC outside of Together's serverless endpoints: there is no open API, pricing, or artifacts for self-hosting. InferenceX results have not been confirmed by a third party.

Sources

Author

Look at AI, editorial team