Agent reliability across repeated runs (tau-bench pass^k)
Share of tau-bench tasks an agent passes on every one of k attempts. The 2024 paper's gpt-4o figures (pass^1 61%, pass^8 under 25%) set the baseline; the 2026 point is the leaderboard leader's pass^1 and pass^4 as tabulated by Automation Anywhere. Sierra's own tau2-bench leaderboard adds pass^1 and pass^4 per model and domain, dated by submission.
Reason through thisThe reading
Reliability still collapses with k - 70% once, 56% four times running - two years after the paper that named the problem.
1 reading drawn from automation_anywhere.tau_bench_leaderboard_1.pass_hat_4.pt, one of 6 series listed under How it is measured; 110 observations in the evidence.
Source: Automation Anywhere blog. Automation Anywhere.
The numbers
| as of | reading | value | record |
|---|---|---|---|
| 2026-05-18 | tau bench leaderboard 1 | 56.2% | reported |
Why it matters
The 70% problem in one number - solving a task once is not solving it reliably. Deployment needs pass^k, not pass^1, and the gap between them is where the normal-technology speed limit lives.
What cuts against it
The 2026 point is vendor-tabulated and the model is unnamed; tau-bench domains are narrow, and on Sierra's leaderboard the best pass^4 ranges from about a third in banking to over nine in ten in telecom, so the domain decides the reading; since March 2026 new models are run on banking only; pass^k penalises benign variation.
How it is measured
- series
automation_anywhere.tau_bench_leaderboard_1.pass_hat_4.pt - series
automation_anywhere.tau_bench_leaderboard_1.pass_hat_1.pt - series
arxiv.tau_bench_retail_gpt4o.pass_hat_1.pt - series
arxiv.tau_bench_retail_gpt4o.pass_hat_8.pt - series
tau2_bench.*.*_pass_4.pt - series
tau2_bench.*.*_pass_1.pt - source arXiv abstract pages · default tier 6 · arXiv; abstract quotation
- source Automation Anywhere blog · default tier 7 · Automation Anywhere; short quotation
- source Sierra tau2-bench leaderboard (pass^k per model and domain) · default tier 1 · Sierra; public leaderboard data, cited with attribution
- proxy benchmark · cadence quarterly · valve invention to product
Normal = under 70% of tasks pass all four runs, so the best agent still fails one attempt in three; fast = 90% or more at k of four or higher, the reliability a workflow can be built on. Between is `emerging`.
Applied to automation_anywhere.tau_bench_leaderboard_1.pass_hat_4.pt.
Latest readings
- 2026-05-18 tau_bench_leaderboard_1 · 56.2%as of 2026-05-18
Why this status, and its history
First scoring. The tau-bench leaderboard leader passes 70.2% of tasks once and 56.2% four times running (Automation Anywhere table, 18 May 2026), inside the normal band; the 2026 point is vendor-tabulated (tier 7) so the status is capped at emerging, with the 2024 paper (gpt-4o pass^1 61%, pass^8 under 25%) as the second source.
- 2026-09-10unmeasured to emergingconf — → 55 · the evaluator (automatic)
First scoring. The tau-bench leaderboard leader passes 70.2% of tasks once and 56.2% four times running (Automation Anywhere table, 18 May 2026), inside the normal band; the 2026 point is vendor-tabulated (tier 7) so the status is capped at emerging, with the 2024 paper (gpt-4o pass^1 61%, pass^8 under 25%) as the second source.
source1
Why this confidence
Rests on 110 observations from Automation Anywhere blog, Sierra tau2-bench leaderboard (pass^k per model and domain), arXiv abstract pages; best evidence tier 1, grade A; current.
Confidence is independent of status: 90–95 multiple strong independent sources; 70–89 good evidence, some ambiguity; 50–69 mixed or hard to operationalise; 0–49 limited or vague evidence.