What is eval saturation?
Eval saturation is the point at which an AI system passes nearly every task in an eval suite, so the suite can no longer show whether a change made the system better or worse.
A capability eval is useful while it has room to move. Once the system passes almost everything, a better prompt and a worse one score the same, and a model upgrade looks like no change at all. The suite has saturated.
Anthropic's agent eval guide lists saturation as something to watch for. It is a sign of progress, but it leaves the team without a measure for the next improvement.
Saturation is not always a problem
A regression suite is meant to sit near 100%; that is its job, and a drop is the signal. Saturation matters for capability evals, whose purpose is to tell versions apart. Anthropic's answer is to graduate solved capability tasks into the regression suite and write harder ones in their place.
Signs a suite has saturated
Scores bunch at the top
Most versions score within a point or two of each other, near the ceiling.
Changes look identical
Prompt edits and model swaps that users clearly notice make no difference to the score.
Judges always pass
An LLM judge returns a pass on almost every trial, which may mean a lenient rubric rather than a strong system.
Production disagrees
Online evals or user feedback show failures the offline suite never catches.
Common causes and fixes
Tasks too easy
Add harder cases from recent production failures and edge cases.
Leakage
Cases copied into prompts or examples; remove them and replace them with unseen cases.
Lenient graders
Tighten the rubric and recalibrate the judge against human labels.
Eval awareness
Anthropic has reported models recognising they are being benchmarked, which is one more reason to prefer real, varied tasks over well-known benchmark items.
How Motiontic's engineers apply it
Motiontic's engineers track the pass rate of each capability suite over time and treat a sustained score near the ceiling as a prompt to move tasks into regression and write new ones from the latest production failures, so the suite keeps measuring the next thing that matters to the client.
Healthy and saturated eval suites compared
| Healthy capability suite | Saturated capability suite | Regression suite | |
|---|---|---|---|
| Pass rate | Well below 100%, climbing | Near 100% | Near 100% by design |
| Tells versions apart | Yes | No | Only by drops |
| Next step | Keep improving | Graduate tasks, write harder ones | Keep running on every change |
Common questions
What is eval saturation?
Eval saturation is when a system passes nearly all tasks in an eval suite, so the suite can no longer distinguish a better version from a worse one.
Is a 100% pass rate a bad sign?
Not for a regression suite, which is meant to pass. For a capability suite it means the suite has stopped measuring progress and needs harder tasks.
How is eval saturation fixed?
Move the solved tasks into the regression suite, add harder cases drawn from real failures, remove any cases that leaked into prompts, and tighten lenient graders.
Related
Eval
Repeatable, scored tests that tell a team whether an AI change made things better or worse.
Regression eval
The suite that catches what a change broke before users do.
Golden dataset
The reviewed, versioned test cases an eval trusts as ground truth.
LLM as a judge
A model grading model output against a rubric, checked against human labels.
AI agent development
Agents that do real work for users, with the evals, traces and guardrails to keep them in production.
Running agents that have to hold up?
Describe the agent, the systems it touches and what it must never get wrong. The first call is with a lab engineer, who will say plainly whether it is ready for production, and whether Motiontic is the right team.