Kids learn with 100m words while LLMs need 15 trillion tokens
mit tech review highlights a data efficiency gap where children master language with ~100m words but llms require trillions of tokens. understanding this discrepancy is critical as the industry faces a potential scarcity of training data by the 2030s.
mit tech review reports on the 'data efficiency gap', highlighting that children learn fluent language on approximately 100 million words, whereas models like meta's llama 3.1 pretrain on 15 trillion tokens—roughly 100,000 times more input. experts warn this brute-force approach is unsustainable, with easily available internet data potentially running dry by the 2030s. researchers aim to reverse-engineer child cognition to create data-efficient ai, which would also benefit minority languages and video learning. this research simultaneously tests linguistic theories, probing whether language acquisition stems from biological instinct or statistical learning constraints.