85 agentic tools audited for security incidents · Talk to the lab →
MotionticBook a call
Glossary

What is eval saturation?

Eval saturation is the point at which an AI system passes nearly every task in an eval suite, so the suite can no longer show whether a change made the system better or worse.

A capability eval is useful while it has room to move. Once the system passes almost everything, a better prompt and a worse one score the same, and a model upgrade looks like no change at all. The suite has saturated.

Anthropic's agent eval guide lists saturation as something to watch for. It is a sign of progress, but it leaves the team without a measure for the next improvement.

Saturation is not always a problem

A regression suite is meant to sit near 100%; that is its job, and a drop is the signal. Saturation matters for capability evals, whose purpose is to tell versions apart. Anthropic's answer is to graduate solved capability tasks into the regression suite and write harder ones in their place.

Signs a suite has saturated

  • Scores bunch at the top

    Most versions score within a point or two of each other, near the ceiling.

  • Changes look identical

    Prompt edits and model swaps that users clearly notice make no difference to the score.

  • Judges always pass

    An LLM judge returns a pass on almost every trial, which may mean a lenient rubric rather than a strong system.

  • Production disagrees

    Online evals or user feedback show failures the offline suite never catches.

Common causes and fixes

  • Tasks too easy

    Add harder cases from recent production failures and edge cases.

  • Leakage

    Cases copied into prompts or examples; remove them and replace them with unseen cases.

  • Lenient graders

    Tighten the rubric and recalibrate the judge against human labels.

  • Eval awareness

    Anthropic has reported models recognising they are being benchmarked, which is one more reason to prefer real, varied tasks over well-known benchmark items.

How Motiontic's engineers apply it

Motiontic's engineers track the pass rate of each capability suite over time and treat a sustained score near the ceiling as a prompt to move tasks into regression and write new ones from the latest production failures, so the suite keeps measuring the next thing that matters to the client.

Healthy and saturated eval suites compared

Healthy and saturated eval suites compared
Healthy capability suiteSaturated capability suiteRegression suite
Pass rateWell below 100%, climbingNear 100%Near 100% by design
Tells versions apartYesNoOnly by drops
Next stepKeep improvingGraduate tasks, write harder onesKeep running on every change

Common questions

What is eval saturation?

Eval saturation is when a system passes nearly all tasks in an eval suite, so the suite can no longer distinguish a better version from a worse one.

Is a 100% pass rate a bad sign?

Not for a regression suite, which is meant to pass. For a capability suite it means the suite has stopped measuring progress and needs harder tasks.

How is eval saturation fixed?

Move the solved tasks into the regression suite, add harder cases drawn from real failures, remove any cases that leaked into prompts, and tighten lenient graders.

Running agents that have to hold up?

Describe the agent, the systems it touches and what it must never get wrong. The first call is with a lab engineer, who will say plainly whether it is ready for production, and whether Motiontic is the right team.