What is a regression eval?
A regression eval is a suite of tasks an AI system already handles correctly, rerun on every prompt, tool, model or data change to confirm the change has not broken behaviour that used to work.
Fixing one failure in an AI system often causes another. A prompt edit that helps refunds can hurt cancellations; a model upgrade that improves reasoning can change output formats. Regression evals exist to catch those side effects.
The suite holds tasks the system has already solved. It should pass at or near 100%, so any drop is a signal that something broke rather than noise.
Capability evals and regression evals
Anthropic's agent eval guide separates the two. Capability evals target what the system cannot yet do reliably and start with a low pass rate that should climb. Regression evals guard what already works and should sit near 100%. As capability tasks are solved, they graduate into the regression suite.
The other labs describe the same idea in their own terms: OpenAI's guidance is to run evals on every change, and Google's ADK runs conformance tests against recorded baselines to catch regressions.
Running regression evals on non-deterministic systems
Several trials per task
Run each task more than once and set a threshold, since one failed run may be noise and one passing run may be luck.
Fixed infrastructure
Anthropic found that container resource limits alone moved agentic coding scores by up to 6 points, so record and pin the eval environment.
Thresholds, not equality
Compare against the last accepted baseline with an agreed tolerance rather than expecting identical output.
Every kind of change
Prompts, tools, retrieval data and model versions all trigger a run, not only code commits.
How Motiontic's engineers apply it
Motiontic's engineers wire the regression suite into CI so it runs on every pull request that touches a prompt, a tool definition or a model setting, and before any model upgrade is switched on. Each production failure that is fixed becomes a new regression task, so the suite grows with the system and stays the client's to own after launch.
Capability evals compared with regression evals
| Capability eval | Regression eval | |
|---|---|---|
| Purpose | Measure progress on hard tasks | Protect behaviour that already works |
| Expected pass rate | Low at first, climbing | At or near 100% |
| A drop means | Usually noise or a harder task mix | Something broke |
| Task source | New and unsolved cases | Solved cases graduated from the capability suite |
| Runs | During development of a feature | On every change, as a release gate |
Common questions
What is a regression eval?
A regression eval is a set of tasks an AI system already passes, rerun after each change to make sure nothing that worked before has broken.
What pass rate should a regression eval have?
At or near 100%. Anthropic's guidance describes regression suites that way, which is what makes any drop meaningful.
When should regression evals run?
On every change to prompts, tools, retrieval data or model version, and before any model upgrade goes live, not only when application code changes.
Related
Eval
Repeatable, scored tests that tell a team whether an AI change made things better or worse.
Golden dataset
The reviewed, versioned test cases an eval trusts as ground truth.
Eval saturation
When every score is near 100% and the eval stops telling anyone anything.
Eval harness
The machinery that runs evals, separate from the agent it tests.
AI agent development
Agents that do real work for users, with the evals, traces and guardrails to keep them in production.
Running agents that have to hold up?
Describe the agent, the systems it touches and what it must never get wrong. The first call is with a lab engineer, who will say plainly whether it is ready for production, and whether Motiontic is the right team.