EMO MoE model keeps 99% performance using only 12.5% of experts
allen institute for ai and uc berkeley developed emo, a mixture-of-experts model that maintains near-full performance while utilizing only 12.5 percent of its total experts.
ai2 and uc berkeley introduced emo, a modular mixture-of-experts language model that assigns experts to specific content domains by enforcing shared expert pools across document boundaries during training. unlike standard moes where experts learn shallow linguistic patterns like prepositions, emo's modules specialize in distinct topics, allowing the removal of unused experts without degrading capabilities. testing showed that keeping just 12.5 percent of experts resulted in a minor three point benchmark drop, whereas a standard moe lost 10 to 15 points in the same scenario. the architecture supports dynamic expert subgroup sizes, enabling efficient on-device deployment and memory savings by loading only relevant experts for specific tasks.