A team from MIT Han Lab, along with engineers from Georgia Tech, UIUC, UC Berkeley, NVIDIA, and OpenAI, has published the Evolving Infra framework — an LLM serving system that adapts to agentic workloads in real time without offline profiling. It separates compute scheduling from KV-cache management: nodes switch between prefill and decode modes in seconds, while the cache is evicted based on context compaction signals from the agent itself. In tests at a scale of up to 256 GB300 GPUs with GLM-5.2 and Kimi K2.6 models, the framework reduced average end-to-end latency by up to 4.1x, tail time-to-first-token (TTFT) by up to 14x, and increased throughput by up to 2.7x compared to NVIDIA Dynamo. The code is released under the MIT license, and the Docker image and benchmarks are already open.

What happened
On November 9, 2026, the preprint of the paper "Evolving Infra: Agent-Adaptive LLM Serving at Scale" (number 2611.12304) appeared on arXiv, with a second version released on November 12. The key architectural decision of the system is the separation of the compute scheduler and the distributed KV-cache manager. The scheduler reconfigures the ratio of nodes between prefill and decode in seconds without restarting the cluster, while the manager uses agent context management signals—primarily dialogue history compaction—as a first-class input: stale cache is evicted, while reusable cache is preserved. Measurements were conducted at a scale of up to 256 GB300 GPUs with GLM-5.2 and Kimi K2.6 models across three scenario classes: coding via Claude Code, math on Putnam, and computer-use in OSWorld. Compared to the production framework NVIDIA Dynamo, average end-to-end latency decreased by up to 4.1x, the P95 TTFT (tail time-to-first-token) metric by up to 14x, and throughput increased by up to 2.7x.
Context
Classic serving clusters are built based on one-time profiling results: some nodes handle prefill, others decode, and these roles are fixed. Agentic workloads break this static nature for two reasons. First, coding agents and computer-use scenarios mix very long prefills—tool call results, command outputs, voluminous code context—with prolonged decoding, so the optimal node ratio changes during operation. Second, the agent harness periodically compresses context: for the serving layer, this means part of the KV-cache loses relevance, while valuable reusable cache is prematurely evicted by standard heuristic policies. Evolving Infra moves context management from the agent layer to the serving layer: the cache manager listens to compaction and sub-agent signals and changes the eviction policy accordingly. Online adaptation without offline profiling is the second methodological pillar of the work, distinguishing it from the common approach of offline cluster tuning for a specific load profile.
Why this matters for the industry
For the industry, this is a concrete mechanism for reducing latency and the cost of agentic inference, not an abstract infrastructure improvement: the tail improvement of up to 14x in P95 TTFT against the production framework NVIDIA Dynamo is achieved by redesigning the layer that operators usually dismiss as unfixable complexity. For teams self-hosting open-weight models like GLM-5.2 and Kimi K2.6, the serving layer becomes a source of direct savings on long agentic runs. Since the code is published under the MIT license, both key mechanisms—dynamic PD-split and KV-cache eviction based on context management signals—are reproducible by anyone, and defensive moats around "smart serving" are blurring. If independent measurements confirm the numbers, similar features are expected to appear in mainstream serving frameworks, along with discussions of a standardized "context events" contract between the agent harness and the cluster; within a couple of years, a transition to co-design of the harness and compute is likely.
Why this matters for users
If you are running LLM agents—Claude Code, Codex, or your own harness—and are hitting limits in response time or inference cost, you can try the system on your own traffic without waiting for vendors. A Docker image, docker.io/weili0234/evolving-infra:v0.1.0, has been published, built for x86_64 (H200, B300) and aarch64 (GB300); your agent connects via an API-compatible router, so no application code rewriting is needed. The repository is structured "agent-natively": deployment and benchmark runs are automated via a coding agent using skills, which significantly lowers the barrier to entry. For those without H200, B300, or GB300, the work is still useful as an open reference for how context compaction and parallel sub-agents affect the KV-cache and compute scheduling, and the published benchmark figures will serve as arguments in negotiations with inference providers.
What is currently unknown / limitations
The work remains a preprint: peer review has not yet been completed, and it is not visible from open materials what exactly changed between version v1 from November 9 and v2 from November 12, 2026. The claimed "up to 14x" is the tail P95 TTFT metric in the best configuration at a scale of up to 256 GB300 against NVIDIA Dynamo; this does not imply near-instant feedback or automatic multiple-fold cost reduction for every agentic inference—a more conservative benchmark is the average end-to-end latency, reduced by up to 4.1x, with throughput up to 2.7x. The impact on other classes of accelerators, other models, and load types not covered in the work has not yet been measured. There are no independent replications yet, although open code and the Docker image facilitate their conduct.
Sources
- Evolving Infra: Agent-Adaptive LLM Serving at Scale (arXiv:2611.12304)
- mit-han-lab/EvolvingInfra — repository with code, README, and benchmarks
Author
Look at AI, editorial team
