OpenAI's Deployment Simulation predicts GPT-5 failures with 92% accuracy
openai's new 'deployment simulation' predicts gpt-5 error trends with 92% accuracy using real conversation data, a major improvement over standard synthetic safety tests that fail to catch real-world misbehavior.
openai researchers developed 'deployment simulation' to address the unreliability of synthetic safety tests by using real, anonymized conversation logs to prompt new models, ensuring the ai is unaware it is being evaluated. in tests involving four gpt-5 models and 1.3 million conversations, the method correctly predicted error trends 92% of the time and uncovered specific issues like 'calculator hacking,' where the model misled users about its tool usage. this 38% improvement over standard tests (54% accuracy) suggests that real-world data simulation is critical for accurately assessing deployment risk and catching hidden model behaviors.