Zyphra Trains Reasoning Model on Full AMD MI300X Stack

IBM Cloud deployment confirms stability for Instinct accelerators during frontier pre-training runs, offering an alternative to Nvidia’s hardware stack.

AI research startup Zyphra has completed end-to-end pre-training for ZAYA1-8B, an open-weights reasoning model, running entirely on AMD hardware inside IBM Cloud infrastructure. The deployment is one of the first documented cases of a large-scale training run executed on AMD MI300X accelerators without touching Nvidia chips or CUDA proprietary libraries.

Steady training runs on AMD MI300X nodes

The workload ran across dedicated clusters scaling up to 1,536 AMD Instinct MI300X accelerators. Nodes communicated over 400 Gbps AMD Pensando Pollara Ethernet adapters, driven by the open-source ROCm software stack. Across the full cycle, the cluster ingested between 14 trillion and 20 trillion tokens. Joint disclosures from Zyphra and AMD indicate the system sustained uninterrupted stretches of 48 to 72 hours without critical hardware faults or node drops.

Sustained operational reliability remains the primary hurdle for non-Nvidia hardware in frontier AI. Engineering teams routinely avoid switching chips for major pre-training runs out of concern that runtime bugs or fabric synchronization bottlenecks will force costly rollbacks. Completing long runs inside IBM Cloud provides verifiable evidence that ROCm and the underlying interconnect have matured enough for production pre-training.

Memory capacity and sparse architecture

ZAYA1-8B contains 8.3 billion total parameters but activates roughly 760 million parameters per token. The architecture pairs a sparse Mixture-of-Experts design (MoE++) with compressed convolutional attention (CCA) to reduce memory footprints during inference. Training was carried out with the Muon optimizer.

Onboard memory directly shaped cluster layout. Each MI300X provides 192 GB of high-bandwidth memory (HBM3), substantially more than the 80 GB or 141 GB typical of competing accelerator generations. That capacity enabled the cluster to handle native 16,000-token sequence lengths within single nodes, eliminating the need for context parallelism across servers and cutting down inter-node network traffic.

Math benchmarks and missing independent audits

Zyphra released the ZAYA1-8B weights under an Apache 2.0 license on Hugging Face and Zyphra Cloud, alongside an unaligned preview of a larger 74-billion-parameter checkpoint. In internal evaluations using test-time compute via Markovian RSA, the team reported scores of 91.9% on the AIME 2025 competition benchmark and 89.6% on HMMT 2025. Those figures rival much larger proprietary models, though they await independent external verification.

Economic comparisons also remain incomplete. Neither AMD nor IBM released third-party audits examining energy draw or total cost of ownership (TCO) per token against equivalent H100 or H200 setups. Even without public cost audits, delivering over 750 PFLOPs of verified capacity under a multi-year cloud contract gives infrastructure planners a working blueprint for diversifying away from single-vendor compute locks.