At the Hot Chips 2026 conference, Microsoft publicly revealed the architecture of Maia 200, the second generation of its in-house AI accelerator for inference in Azure. This is a rare case where a hyperscaler shows honest engineering numbers for a chip rather than marketing promises: the design is based on Software Defined Local Access (SDLA), where the data flow is fixed at the compilation stage, and communication between thousands of chips goes over a single Ethernet rather than a separate scale-up interconnect.

image

What happened

Microsoft presented Maia 200 at Hot Chips 2026 and laid out a detailed picture of the chip: TSMC 3 nm process, 140 billion transistors on an 820 mm² die, and a maximum thermal design power of 750 W. Memory consists of six HBM3e stacks with a total bandwidth of 7 TB/s, and peak performance reaches 10,000 TFLOPS in the 4-bit FP4 format created for inference. The architectural core is Software Defined Local Access: the data path is defined in software and locked in at the compilation stage, while data access remains local within compute elements. Each basic tile contains a tensor block TTU, a SIMD processor TVP, a control block TCP, a DMA block, and an L1 cache. Scaling is provided only in the scale-up plane and only over a single Ethernet: a proprietary NIC and protocol, eight Ethernet lines per chip grouped into four network planes, a reference configuration of 128 racks and 6,000 chips, and collective operations performed through the MCCL (Microsoft Collective Communication Library). In the published microbenchmarks, GEMM operates at the theoretical limit (roofline), BF16 AllReduce shows about 1.3 TB/s, All2All up to 655 GB/s, and collective operations complete in a single hop.

Context

Cloud providers have historically had two working paths in AI computing: buying NVIDIA GPUs or building fully custom ASICs along the lines of TPU. Maia 200 is built as a third path between them: not a universal GPU, but also not a closed research chip. The bet is specifically on inference, which is the main cost item in Azure, and primarily on MoE models — separating prefill and decode, expert parallelism, and minimal memory traffic determine the economics of these workloads, which is why the SDLA architecture keeps data within local tiles. Detailing an accelerator at Hot Chips is a familiar industry signal that the chip is being prepared for real data-center deployments.

Why this matters for the industry

Maia 200 is the first public slice of a hyperscaler's non-NVIDIA accelerator architecture with verifiable numbers, and the ecosystem can use it as a reference for calibrating expectations of inference hardware. Abandoning a separate scale-up interconnect in favor of a single Ethernet fabric is a bold architectural bet: if it is confirmed in production, the approach will become a model for the next generation of accelerators, and MCCL collective communications on a single fabric are contenders for the status of an industry standard. Specific microbenchmarks instead of marketing phrases like “up to N TFLOPS” speak to the maturity of compiler-hardware co-design. If deployment in Azure begins, Maia 200 will become a direct competitor to NVIDIA in cloud inference workloads and a factor pressuring GPU prices and their energy-efficiency requirements.

Why this matters for users

There is no new API, model, or service for developers in this announcement: Maia 200 is internal Azure infrastructure, with no GA, pricing, or external access. The impact will be indirect: if the chip becomes the backbone of Azure inference, MoE model inference may become cheaper and faster, and with it so will what Copilot and OpenAI models run on in Azure. The material is already useful for planning: the announcement adjusts expectations for token cost and latency, and those who calculate the economics of products on Azure inference can revise their assumptions about token price per task. For engineers and researchers, this is a rare public source of honest numbers on a non-NVIDIA accelerator that is usually hidden behind marketing.

What is still unknown / limitations

The materials do not include the cost of Maia 200, yield, or price per token, and only microbenchmarks have been published: there is no data on end-to-end performance of full models. Reliability and latency at the reference scale of 6,000 chips have not been publicly confirmed. GA dates, pricing, and access for third-party users have not been announced. Forecasts about price pressure on LLM inference and the idea that inference optimizations are ceasing to be a competitive advantage are business interpretations, not technical statements.

Sources

Author

Look at AI, editorial team