Open Athena, a joint project with Stanford, published a detailed breakdown of the Marin 535B-A23B architecture — a 535-billion parameter MoE model, of which 23 billion are active at each step. On November 5, 2026, the model completed pretraining on 18 trillion tokens, which ran on 768 GB200 GPUs. The final Paloma macro-loss was 1.982 against a pre-registered forecast of 2.039, and no loss spikes occurred throughout the run. The publication reveals key architectural decisions and engineering techniques applied during training.
What happened
The breakdown in the Open Athena blog describes the model down to specific engineering decisions. The foundation is Latent MoE: 384 latent experts, of which 8 are activated, plus 2 shared experts, with a 2x compression of representations. Expert Parallelism has been moved to a ragged all-to-all scheme, and the drop rate decreased from 5.5–7.1% to approximately 0.01–0.14%. The optimizer is proprietary — Hyperball, where MuonH handles matrices and AdamH handles lm_head. Among other components are Exclusive Self Attention (XSA), Short Convolutions (SConv) with kernel 4, QK RMSNorm with a fixed multiplier of 1.3, and top-k quantile balancing on raw logits for router stabilization. Context was expanded in stages during training from 4k to 16k and then to 65k tokens.
Context
The forecast for the final loss was recorded before the start of the run — a practice borrowed from science, where a hypothesis is fixed in advance so that the result cannot be fitted after the fact. Open Athena has transferred this approach to model scaling and conducted training openly: the run was accompanied by public reporting and live streams, and Marin's experiments, data, and infrastructure are published. The breakdown emphasizes that the architecture was designed with a focus on MFU (Model FLOPs Utilization), i.e., useful compute load — hence the attention to the computational efficiency of each decision.
Why this matters for the industry
For the industry, this is a rare public playbook for a full-cycle large-scale MoE pretraining: numbers, graphs, and decisions are laid out in a way that allows techniques to be transferred to in-house training pipelines and run planning immediately, without waiting for a product announcement. The combination of Expert Parallelism and Latent MoE is estimated by the authors to provide about a 1.3x gain in computational efficiency, and measurable metrics like drop rates allow third-party teams to verify techniques on their own at smaller scales. A separate signal is the framing itself: pretraining at a scale close to frontier is shown as a predictable engineering task. However, the lever of the event is informational rather than market: the publication is dedicated to pretraining and does not contain serving data, so it does not currently affect inference or product decisions.
Why this matters for users
For readers, this is an opportunity to look into processes that usually remain inside frontier labs: the publication shows not only the final result but also how engineering decisions were made during training. All numbers and graphs — the loss curve for the entire run, drop rates, scaling ladder — are laid out in the official Open Athena blog, so the material can be used as a course on the workings of modern pretraining. For those studying model training, it will provide an understanding of the trade-offs behind each architectural choice.
What is still unknown / limitations
Conclusions are limited by several constraints. The accuracy of the forecast proves the predictability of this specific run, not the statistical reliability of the pattern: this is one self-published run, and independent replications are needed, at least at smaller scales. Paloma macro-loss is a training proxy; there are no downstream benchmarks in the sources, so judgments about the quality of the finished model are premature. Several new components were implemented simultaneously without ablations in the presentation, so attributing the claimed increase in computational efficiency to specific decisions based on a single breakdown is impossible. Finally, there are no comparisons with other runs of the same scale and budget in the sources.
Sources
Author
Look at AI, editorial team