为什么小型模型无法掌握 GPT-4 级别精通的罕见技能

researchers from anthropic and stanford identified why larger language models (like olmo 1b+) learn rare tasks while smaller models fail: frequent training updates crowd out sparse signals before they can stick. small models enter a cycle where rare examples are learned and immediately erased by dominant, common tasks, whereas large models retain them once common tasks are saturated. experiments with olmo models ranging from 4m to 4b parameters showed that smaller models never achieve 'grokking' on rare modular arithmetic tasks regardless of training duration. this challenges the assumption that more data fixes performance gaps, suggesting that increasing the frequency of rare data may be more efficient than massive scaling.