OpenAI's Perplexity Metric for AI Progress
openai chief scientist jakub pachocki argues perplexity is key to understanding ai advances, proposes it as a metric for shared safety standards.
jakub pachocki, openai's chief scientist, believes that tracking progress on training better compressors, measured by perplexity, is key to understanding advances in ai. he argues that perplexity, or cross-entropy loss, should be used as a metric to compare models and inform shared safety standards across the industry. openai has shared its internal perplexity measurements, showing progress from gpt-3.5 to gpt-4, with lower perplexity values indicating better predictions. pachocki notes that this metric has remained constant over the years, despite rapid ai advancements.