Hugging Face releases EMO MoE with emergent modularity

Hugging Face released EMO (Emergent Modularity), a 1B-active, 14B-total-parameter Mixture-of-Experts (MoE) model. Unlike standard MoEs, EMO pretrains with a document-level routing constraint, allowing modular structure to emerge without human-defined semantic labels. This architecture enables users to activate only 12.5% of the total experts for specific tasks like math or code while retaining near full-model performance, whereas standard MoEs degrade significantly under subset selection. The model remains a strong general-purpose model when all 128 experts are active.