EMO MoE 모델, 전체 전문가 중 12.5%만 사용해 99% 성능 유지
allen institute for ai와 uc berkeley가 전체 전문가 중 12.5%만 활용하면서도 거의 전체 수준의 성능을 유지하는 mixture-of-experts 모델인 emo를 개발했다.
ai2 and uc berkeley introduced emo, a modular mixture-of-experts language model that assigns experts to specific content domains by enforcing shared expert pools across document boundaries during training. unlike standard moes where experts learn shallow linguistic patterns like prepositions, emo's modules specialize in distinct topics, allowing the removal of unused experts without degrading capabilities. testing showed that keeping just 12.5 percent of experts resulted in a minor three point benchmark drop, whereas a standard moe lost 10 to 15 points in the same scenario. the architecture supports dynamic expert subgroup sizes, enabling efficient on-device deployment and memory savings by loading only relevant experts for specific tasks.