💻 LLM Serving Adapted to Agentic Workloads

MIT Han Lab, Georgia Tech, UIUC, UC Berkeley, NVIDIA, and OpenAI published Evolving Infra on arXiv — an LLM serving framework that adapts online to agentic workloads. Nodes switch between prefill and decode in seconds, while the KV-manager uses agent signals to evict stale cache and preserve reusable data. On 256 GB300 GPUs with GLM-5.2 and Kimi K2.6, average latency is up to 4.1x lower, P95 TTFT is up to 14x lower, and throughput is up to 2.7x higher compared to NVIDIA Dynamo.

🌍 Agentic workloads break static prefill-decode clusters: compaction invalidates the KV-cache, and reusable cache is evicted prematurely — context management moves to the cluster layer.

👤 Running agents and hitting latency or inference cost limits — the code is open under the MIT license, with a Docker image and an API-compatible router.

Source 1: https://arxiv.org/abs/2611.12304 Source 2: https://github.com/mit-han-lab/EvolvingInfra