Advancing Microsoft’s AI Infrastructure

Microsoft has officially unveiled the Maia 300, the latest iteration of its custom Azure silicon. This announcement marks a significant step forward as the company prepares to expand its internal AI infrastructure to support massive, multi-gigawatt deployments.

Scheduled for deployment starting in 2027, the Maia 300 is built with an "inference-first" philosophy. Microsoft reports that the chip will deliver 40% more tokens per megawatt compared to leading GPU-based systems. It is specifically optimized for complex tasks, including long-context reasoning, responsive conversational AI, and sustained multi-step agent operations.


Technical Innovations and Architecture

The Maia 300 introduces a sophisticated multi-chiplet architecture that segregates compute, memory-controller, and input/output functions. Key technical highlights include:

  • High-Performance Memory: The chip utilizes eight stacks of HBM4 memory, boasting a peak bandwidth of 18TB/s within a 950W thermal design power.
  • Efficient Dataflow: A software-defined architecture combined with AI-assisted kernel optimization ensures data remains localized to the system components that require it, minimizing movement.
  • Advanced Manufacturing: The compute chiplets will be fabricated using TSMC’s N2P process.
  • Format Optimization: Support for low-precision formats, such as enhanced MXFP4, allows the system to manage larger context windows while maintaining power efficiency.

Scalability and Integration

Microsoft has designed the networking capabilities to function as an integral part of the dataflow. By utilizing high-bandwidth connections between accelerators for tensor parallelism alongside custom AI transport based on Ethernet, the architecture is capable of scaling from individual racks to entire data centers.

A single Maia 300 rack will house 96 accelerators, providing 27.6TB of HBM capacity and an aggregate bandwidth of 1.7 petabytes per second. According to the company, this density will enable more simultaneous AI interactions and the processing of more demanding workloads per rack.


Broad Impact on Services

The Maia 300 is expected to bolster a wide array of Microsoft products. The company plans to leverage this technology for everything from agentic coding in GitHub to large-scale model serving via Microsoft Foundry. Additionally, the chip is set to enhance the performance and data generation capabilities for its internal AI division, directly benefiting end-user tools such as:

«We anticipate that the improved inference capabilities will lead to significant performance gains across our key products, including Copilot, Cowork, and Microsoft Teams.»

This initiative builds on the success of the Maia 200, which is currently handling production workloads, including those for Microsoft 365 Copilot and various models developed by the Microsoft AI team.