85 agentic tools audited for security incidents · Talk to the lab →
MotionticBook a call
Glossary

What is a capability eval?

A capability eval is a set of test tasks built to measure what an AI model or agent can do well, deliberately starting at a low pass rate so the team has a hill to climb.

Most eval suites answer one question: did the last change break anything? A capability eval answers a different one: what can the agent not yet do, and is it getting closer? Anthropic's January 2026 guide to agent evals separates the two explicitly. Capability evals, which it also calls quality evals, should start at a low pass rate; regression evals should sit near 100%.

A suite where everything passes cannot show progress, so a capability eval is designed to be failed at first. It is where a new feature is specified before the agent can perform it, which Anthropic describes as eval-driven development: write the tasks, then iterate until the agent passes them.

How a capability eval is built

  • Tasks the agent should learn

    Drawn from real failures, feature requests and edge cases. Anthropic's guidance is that 20 to 50 simple tasks from real failures are enough to start, rather than waiting for hundreds.

  • Clear success criteria

    Each task states its inputs and what success looks like, precisely enough that two engineers would grade the same attempt the same way.

  • Several trials per task

    Agent output varies from run to run, so each task is attempted more than once and reported as pass@k, pass^k or both.

  • Graders matched to the task

    Code checks where the outcome can be verified, model-based graders for open-ended output, and human review to calibrate the model graders.

Graduation, and what a 0% score means

After launch, capability tasks that reach a high pass rate graduate into the regression suite, which runs continuously. The question changes from whether the agent can do the task at all to whether it still does it reliably. Harder tasks replace the graduates, so the capability suite keeps a hill to climb instead of saturating.

A task that scores 0% across many trials is a warning as much as a result. Anthropic notes that 0% pass@100 most often signals a broken task, not an incapable agent. Its example: Claude Opus 4.5 first scored 42% on CORE-Bench, and fixing grading bugs, clarifying ambiguous specs and using a less constrained scaffold took it to 95%.

The other meaning: dangerous-capability evals

In frontier AI safety, capability evals also test for abilities a lab does not want a model to have. Google DeepMind's paper Evaluating Frontier Models for Dangerous Capabilities (March 2024) covers persuasion and deception, cyber-security, self-proliferation and self-reasoning. The mechanics are the same, tasks, trials and graders, but the goal is inverted: a low score is the result the lab hopes for.

Teams building product agents almost always mean the first sense: a suite that tracks progress on the work the agent is being built to do.

How Motiontic's engineers apply it

Motiontic's engineers write capability tasks for a feature before building it, from the client's real cases, so progress is measured from the first week. Transcripts are read before any score is trusted, and tasks that pass consistently move into the regression suite that runs in CI.

Capability eval compared with regression eval

Capability eval compared with regression eval
Capability evalRegression eval
QuestionWhat can the agent not do well yet?Does it still do what it used to?
Expected pass rateLow at first, rising over timeNear 100%
When it runsWhile a feature or model change is developedOn every change, in CI
A failure meansWork still to do, or a broken taskA regression that blocks the release
LifecycleTasks graduate out once they pass reliablyGrows as tasks graduate in

Common questions

What is the difference between a capability eval and a regression eval?

A capability eval measures what an agent cannot yet do reliably and starts with a low pass rate. A regression eval checks that solved tasks stay solved and should pass close to 100% of the time.

How many tasks does a capability eval need?

Anthropic's guidance is to start with 20 to 50 simple tasks drawn from real failures rather than waiting for hundreds. Tasks are added as the agent improves and as new failures appear.

When does a capability task move to the regression suite?

When it passes consistently across trials. At that point it protects a solved behaviour, and a harder task takes its place in the capability suite.

Is a capability eval the same as a benchmark?

A public benchmark is a shared capability eval used to compare models. A team's own capability eval uses its own tasks and success criteria, so it measures the agent on the work it will actually do.

Running agents that have to hold up?

Describe the agent, the systems it touches and what it must never get wrong. The first call is with a lab engineer, who will say plainly whether it is ready for production, and whether Motiontic is the right team.