Hugging Face, 창발적 모듈성을 갖춘 EMO MoE 공개
Hugging Face가 데이터로부터 모듈성이 창발하는 140억(14B) 파라미터 MoE 모델 EMO를 공개했습니다. 이 모델은 작업당 전문가(experts)의 12.5%만 사용합니다.
Hugging Face released EMO (Emergent Modularity), a 1B-active, 14B-total-parameter Mixture-of-Experts (MoE) model. Unlike standard MoEs, EMO pretrains with a document-level routing constraint, allowing modular structure to emerge without human-defined semantic labels. This architecture enables users to activate only 12.5% of the total experts for specific tasks like math or code while retaining near full-model performance, whereas standard MoEs degrade significantly under subset selection. The model remains a strong general-purpose model when all 128 experts are active.