gpt-4급 모델이 완벽히 해내던 희귀 작업에서 소형 모델이 실패하는 이유
Anthropic과 스탠퍼드 연구진은 대형 모델이 빈번한 작업이 포화 상태에 이른 후에야 희귀 스킬을 습득함을 입증했으며, 이는 소형 모델을 단순히 더 오래 학습시키는 것이 무의미하다는 것을 보여줍니다.
researchers from anthropic and stanford identified why larger language models (like olmo 1b+) learn rare tasks while smaller models fail: frequent training updates crowd out sparse signals before they can stick. small models enter a cycle where rare examples are learned and immediately erased by dominant, common tasks, whereas large models retain them once common tasks are saturated. experiments with olmo models ranging from 4m to 4b parameters showed that smaller models never achieve 'grokking' on rare modular arithmetic tasks regardless of training duration. this challenges the assumption that more data fixes performance gaps, suggesting that increasing the frequency of rare data may be more efficient than massive scaling.