Periodic Labs, a company developing AI for high-throughput experiments in superconductor and magnet discovery, published a detailed breakdown of its internal AI infrastructure for scientific reinforcement learning on September 15, 2026, where a single RL rollout can last over an hour. The final training run, started from open weights, peaked at 1300 H200 GPUs during midtraining and RL stages, with a claimed increase in training throughput relative to the Megatron baseline on the same GPUs reaching 4.1x. The company has already returned key optimizations to open upstreams, although all cited figures remain self-reported.
What happened
The post “AI Infrastructure at Periodic” describes a stack built on Megatron, SGLang, Miles, and Ray, and confirms it with measurements. Decoding of a 1T-parameter model is accelerated 2.5x — from 10 to 25 tok/s per request. Rebuilding a checkpoint from Megatron format to Hugging Face was reduced from 30 minutes to 1 minute. The company’s own sandbox system, pbox, built on gVisor over Slurm-on-Kubernetes, shows 4.5x lower p50 latency for transferring 1 MiB.
Context
Periodic Labs operates at the intersection of AI and experimental science: using high-throughput experiments, the company searches for superconductors and magnets. Scientific reinforcement learning in such an environment is structured differently than training ordinary chat models: the agent performs long multi-step sessions with tool calls, and bottlenecks shift from the training itself to inference, sandbox isolation, and managing checkpoints for trillion-parameter models. It is precisely on this class of workload that standard solutions — Megatron as the training base and hosted sandbox providers — hit a ceiling, so components have to be customized. The breakdown mentions WideEP, decoupling of prefill and decode, delta router replay, and chunked optimizer offload in Megatron-LM; the point of the decoupling is that context processing and token generation can be scaled separately.
Why this matters for the industry
For the industry, the main signal is that scientific RL with multi-hour rollouts and tool calls has become a separate class of infrastructure tasks with its own set of patterns. The value for builders lies in portable artifacts: delta router replay was sent to SGLang PR #24851, and fast resharding — to Miles PR #1371; if the changes are accepted and stabilized, the benefit will go to all users of these frameworks without copying Periodic Labs’ internal stack. A separate benchmark for GPU economics — maintaining 95%+ cluster utilization through sharing with simulation workloads: for teams with H200-level clusters, this is a direct way to reduce idle time of expensive cards. Finally, the approach with its own sandbox isolation shows that agentic workloads with frequent tool calls can be handled in-house, not only through hosted providers.
Why this matters for users
For the engineer-reader, the post is useful as a compact reference for modern RL stack optimizations with verifiable artifacts: both mentioned PRs can be opened in the upstream and the code evaluated, instead of taking the numbers at face value. The material shows how the trillion-parameter Kimi K2.6 is trained with full parameters on 64 H200s at 64K context, reducing peak memory from 320 to 132 GB — a concrete benchmark for teams that don’t have thousands of GPUs. Patterns can also be adopted, not just code: decoupling of prefill and decode, chunked optimizer offload, and gVisor sandbox isolation. The Periodic Labs stack itself cannot be deployed directly: it is internal, without API, pricing, or documentation for third parties, so the value of the material is currently referential, not product-based.
What is still unknown / limitations
All key metrics are self-reported by Periodic Labs based on its own measurements: the comparison conditions with the Megatron baseline (configuration, parallelism, context, batch) are not disclosed in the available materials, so the claimed multipliers should be considered a hypothesis until independent reproduction. The second pbox metric — a 3.3x gain — is cited without specifying which exact metric it refers to. The claim that specialized models Pareto-dominate GPT-6 Astra and Claude Fable 5.1 is based on the company’s own X-ray diffraction evaluations without a published protocol and independent evaluation. Upstream PRs are not yet merged and stabilized, and community reaction is minimal: the Hacker News discussion has 2 points and 0 comments.
Sources
Author
Look at AI, editorial team