Back to Articles a]:hidden"> Your Agent Aced the Task. Will It Do It Again? Enterprise Article Published September 15, 2026 Upvote 119 +113 Evelyn Duesterwald evduester Follow ibm-research Lilian Ngweta lilianngweta Follow ibm-research Vatche Isahagian Vatche Follow ibm-research Jayaram Radhakrishnan jayaramkr Follow ibm-research Vinod Muthusamy vinodmut Follow ibm-research Gaodan Fang gaodan-fang Follow ibm-research Ashwath Vaithinathan Aravindan ashwath-vaithina Follow ibm-research Punleuk Oum illeatmyhat Follow ibm-research G Thomas gsthomasx Follow ibm-research Merve Unuvar mrvnvr Follow ibm-research Ayhan Sebin ayhansebin Follow ibm-research Michał Ulewicz Michal Follow ibm-research :last-child]:mb-0"> The Metric Almost Nobody Reports Why Agents Flip: Sharp Decisions vs. Flat Ones Diagnose, Then Fix Results: Reducing the Gap Without Losing Accuracy The guidelines generalize — they aren't patching one trajectory If You're Shipping an Agent Try It Appendix: Understanding the Metrics Linked artifacts / references Your agent works in rehearsal, but during the live demo, it takes a different path and fails the same task
That is embarrassing onstage. In production, it is a reliability problem: a workflow that succeeded once may fail the next time a user makes the same request. For mission-critical work, such as reconciling a financial transaction or checking a contract for an obligation, that can be a showstopper
Most benchmarks hide this variability behind an average. On AppWorld, a ReAct agent using GPT-4.1 succeeded on 77.4% of runs across five repetitions. But it succeeded in all five runs for only 53.0% of tasks — a 24.4-point consistency gap
