EMO混合专家模型仅使用12.5%专家即可保持99%性能

ai2 and uc berkeley introduced emo, a modular mixture-of-experts language model that assigns experts to specific content domains by enforcing shared expert pools across document boundaries during training. unlike standard moes where experts learn shallow linguistic patterns like prepositions, emo's modules specialize in distinct topics, allowing the removal of unused experts without degrading capabilities. testing showed that keeping just 12.5 percent of experts resulted in a minor three point benchmark drop, whereas a standard moe lost 10 to 15 points in the same scenario. the architecture supports dynamic expert subgroup sizes, enabling efficient on-device deployment and memory savings by loading only relevant experts for specific tasks.