GPT-4.1 agent consistency gap hits 24.4 points on AppWorld
a react agent using gpt-4.1 succeeded on 77.4% of runs but only 53.0% of tasks consistently, affecting mission-critical work reliability.
on appworld, a react agent using gpt-4.1 achieved a 77.4% success rate across five repetitions but only succeeded consistently on 53.0% of tasks, indicating a 24.4-point consistency gap. this issue affects mission-critical workflows, where reliability is crucial. to address this, researchers developed the consistency analyzer and a new guideline type in altk-evolve, which halves the consistency gap without impacting average accuracy. the method involves analyzing an agent's past trajectories to identify and improve decision-making consistency.