gpt-4クラスが容易に解けたレアタスクで小型モデルが失敗する理由
AnthropicとStanfordの研究者らは、大規模モデルがレアなスキルを習得するのは頻出タスクが飽和した後であることを実証し、小型モデルを単に長く訓練しても無駄であることを証明しました。
researchers from anthropic and stanford identified why larger language models (like olmo 1b+) learn rare tasks while smaller models fail: frequent training updates crowd out sparse signals before they can stick. small models enter a cycle where rare examples are learned and immediately erased by dominant, common tasks, whereas large models retain them once common tasks are saturated. experiments with olmo models ranging from 4m to 4b parameters showed that smaller models never achieve 'grokking' on rare modular arithmetic tasks regardless of training duration. this challenges the assumption that more data fixes performance gaps, suggesting that increasing the frequency of rare data may be more efficient than massive scaling.