Frontier model evaluations are broken: is the 'harness' to blame?
openai argues that independent safety evaluations fail without accounting for the 'harness'—the tools and workflows that determine a model's true performance.
openai published a framework establishing new standards for independent frontier model evaluations, shifting focus from raw model outputs to the 'harness' of tools and workflows that enables agency. the post categorizes claims into capability elicitation, safeguard performance, and comparison, while warning that traditional evaluations miss critical behaviors like reward hacking or sandbagging. this signals an industry-wide move towards more rigorous, system-level testing protocols that go beyond chat prompts to assess how models act over multi-step trajectories.