Open SourceMedium importance
Hugging Face releases 'Ultimate MoE Training Handbook' after ~800k GPU-hours
On 11/05/2026 Thomas Wolf (@Thom_Wolf) announced on X 'The Ultimate MoE Training Handbook' (HF Space huggingface/moe-blogpost): a free open guide (~70k words, 55 interactive figures, runnable code) on training Mixture-of-Experts with Megatron-LM, based on ~800k GH200 GPU-hours on the Alps cluster (CSCS). Cited context: 18 of the top 20 current open models are MoE. Covers expert sizing, all-to-all, routers, aux loss vs DeepSeek-style bias loss-free, upcycling vs from-scratch, serving. Recommended scripts and flags; gallery at huggingface/moe-gallery.
Published in edition no. 13 of Noticiário do Joãozinho ·
Translated by Joãozinho AI from the original Brazilian Portuguese edition. Facts, numbers and sources are the same as in the original; quotes, video titles and source names stay in their original language. Read the original in Portuguese
Original source
Hugging Face / Thomas Wolf (X)X @Thom_WolfFigure gallery
https://huggingface.co/spaces/huggingface/moe-blogpost