Hugging Face unveils Olmo-core 3 for scalable MoE training
hugging face releases olmo-core 3, an open, scalable training infrastructure for large mixture-of-experts models, to improve efficiency and reduce costs. this affects ai researchers and developers.
hugging face has released olmo-core 3, a major update to its framework for developing large language models. the new system features a redesigned open mixture-of-experts training infrastructure, allowing for scalable training into the trillion-parameter range. olmo-core 3 uses techniques such as expert parallelism, pipeline parallelism, and a distributed optimizer to distribute large models across gpu clusters efficiently. in benchmarks, the system showed a significant increase in throughput, processing 52,000 tokens per second per gpu. this upgrade aims to reduce the computational costs and energy use associated with training large ai models, making advanced model development more accessible to researchers and smaller labs.