Hugging Face adds first-class Mixture-of-Experts support
hugging face refactored transformers to support mixture-of-experts (moe) models natively. the update addresses weight loading, device placement, and distributed execution for sparse architectures like deepseek r1 and mixtral. this allows models to use fewer active parameters during inference while maintaining total capacity, significantly improving compute efficiency and speed.
Hugging Face has refactored the Transformers library to support Mixture-of-Experts (MoE) architectures as first-class citizens. The update redesigns core pipelines—including weight loading, device placement, and distributed execution—to handle sparse models that previously strained tools built for dense networks.
MoEs decouple total parameter count from inference speed by activating only a subset of experts per token. For example, the open gpt-oss-20b model runs at approximately 115 tokens per second on an M3 Ultra by activating only 3.6B parameters out of 21B total. Recent high-profile MoE releases include DeepSeek R1, Qwen 3.5, and MiniMax M2.