EMO MoEモデル、エキスパートの12.5%のみで99%の性能を維持
Allen Institute for AIとUC Berkeleyは、全エキスパートの12.5%のみを活用しながらほぼフルの性能を維持するMixture-of-Expertsモデル「EMO」を開発した。
ai2 and uc berkeley introduced emo, a modular mixture-of-experts language model that assigns experts to specific content domains by enforcing shared expert pools across document boundaries during training. unlike standard moes where experts learn shallow linguistic patterns like prepositions, emo's modules specialize in distinct topics, allowing the removal of unused experts without degrading capabilities. testing showed that keeping just 12.5 percent of experts resulted in a minor three point benchmark drop, whereas a standard moe lost 10 to 15 points in the same scenario. the architecture supports dynamic expert subgroup sizes, enabling efficient on-device deployment and memory savings by loading only relevant experts for specific tasks.