What are pass@k and pass^k?
pass@k is the probability that at least one of k attempts at a task succeeds, while pass^k is the probability that all k attempts succeed; the first measures capability and the second measures reliability.
Agents do not give the same result every run. A task that succeeds three times out of four can look solved or broken depending on how the runs are counted. pass@k and pass^k are two ways of counting, and they answer different questions.
pass@k is generous: it counts a task as solved if any of k tries works, which suits a setting where a person or a test can pick the good attempt. pass^k is strict: every one of k tries must work, which is closer to what a customer sees when the agent runs on every request.
How the two are calculated
If each trial succeeds independently with probability p, then pass@k equals 1 minus (1 minus p) to the power k, and pass^k equals p to the power k. The two move in opposite directions: as k grows, pass@k rises towards 100% and pass^k falls towards zero.
Anthropic's agent eval guide gives the example of a 75% per-trial success rate: pass^3 is about 42%. On the same numbers, pass@3 is about 98%. One agent, one task, and the headline figure ranges from 42% to 98% depending on the metric.
Where the metrics come from
pass@k comes from code generation benchmarks, where a model writes several candidate programs and a task counts as solved if any of them passes the unit tests. pass^k was proposed in the τ-bench paper (June 2024) specifically to measure how reliably an agent behaves over repeated trials of the same task.
Anthropic's January 2026 guide recommends tracking both, and using pass^k where customer-facing consistency matters.
Which one to report
Exploring capability
pass@k shows whether a task is within reach at all, which helps when choosing a model or deciding whether an approach is worth pursuing.
Shipping to customers
pass^k shows how often every user would get a correct result, which is the number a support or finance workflow has to live with.
Always alongside
Report the per-trial success rate and the value of k next to either metric; without them the figure cannot be compared.
How Motiontic's engineers apply it
Motiontic's engineers run each eval task several times and report pass^k for any workflow a customer or employee relies on without checking. A task with a high pass@k and a low pass^k is a sign the agent can do the work but not consistently, which usually points to ambiguous instructions, a fragile tool or missing context rather than a weak model.
pass@k compared with pass^k
| pass@k | pass^k | |
|---|---|---|
| Question | Does at least one of k attempts succeed? | Do all k attempts succeed? |
| Measures | Capability | Reliability and consistency |
| As k grows | Rises towards 100% | Falls towards zero |
| At 75% per trial, k = 3 | About 98% | About 42% |
| Fits | Research, model selection, human-picked outputs | Customer-facing and unattended agents |
Common questions
What does pass@k mean?
pass@k is the share of tasks where at least one of k attempts succeeds. It shows what a model or agent is capable of when several tries are allowed.
What does pass^k mean?
pass^k is the share of tasks where all k attempts succeed. It shows how consistently an agent gets a task right, which matters when every run reaches a user.
Which metric should a production agent be judged on?
For agents that act without a person checking each result, pass^k is the more honest figure. Anthropic's eval guidance recommends it where customer-facing consistency matters.
Why does pass^k fall so quickly?
Because every attempt must succeed, the per-trial rate is multiplied by itself k times. At 90% per trial, pass^5 is about 59%, so small gains in per-trial reliability make a large difference.
Related
Eval
Repeatable, scored tests that tell a team whether an AI change made things better or worse.
Regression eval
The suite that catches what a change broke before users do.
Golden dataset
The reviewed, versioned test cases an eval trusts as ground truth.
Eval harness
The machinery that runs evals, separate from the agent it tests.
AI agent development
Agents that do real work for users, with the evals, traces and guardrails to keep them in production.
Running agents that have to hold up?
Describe the agent, the systems it touches and what it must never get wrong. The first call is with a lab engineer, who will say plainly whether it is ready for production, and whether Motiontic is the right team.