85 agentic tools audited for security incidents · Talk to the lab →
MotionticBook a call
Glossary

What is a grader in AI evals?

A grader, in AI evals, is the logic that scores some aspect of an agent's or model's attempt at a task, using code checks, a model following a rubric, or a human reviewer.

An eval task says what should happen; the grader decides whether it did. Anthropic's January 2026 guide defines a grader as logic that scores some aspect of performance, and notes that one task can carry several graders, each with several assertions: was the customer verified first, was the refund issued, was the reply grounded in the tool results.

Graders are where many eval results go wrong. A score is only as good as the check behind it, so a grader needs testing and calibration of its own, and someone has to read the transcripts it judged.

Three kinds of grader

  • Code-based

    String matching, unit tests, static analysis, checks on the state of the environment and checks on which tools were called. Fast, cheap, reproducible and easy to debug, but brittle when a valid answer takes an unexpected form, and weak on subjective work.

  • Model-based

    Rubric scoring, natural-language assertions, pairwise comparison, reference-based checks and multi-judge consensus. Flexible and able to handle open-ended output, but non-deterministic, more expensive than code and in need of calibration against human judgement.

  • Human

    Subject-matter-expert review, spot-check sampling, A/B tests and inter-annotator agreement. The gold standard, and the source used to calibrate model graders, but slow and expensive at scale.

Grade the outcome, give partial credit

Anthropic's advice is to grade what the agent produced rather than the exact sequence of steps it took, because rigid step-order checks penalise valid solutions no one anticipated. The outcome is the final state of the environment: whether a reservation exists in the database, not whether the agent said it made one.

For multi-step tasks, partial credit separates near misses from early failures. An agent that verifies the customer and identifies the problem but fails to issue the refund has done better than one that fails at the first step, and a binary grader hides the difference. Where the path itself is a requirement, such as a compliance check that must happen first, trajectory evaluation adds checks on the steps.

How graders fail

  • Too strict

    Anthropic's CORE-Bench example: a grader rejected 96.12 when the expected value was 96.124991. Fixing grading bugs and ambiguous specs, with a less constrained scaffold, moved Claude Opus 4.5 from 42% to 95%.

  • Misconfigured

    METR found tasks that required exceeding a stated score threshold, which penalised models that followed the instructions and rewarded those that ignored them.

  • Checking the claim, not the state

    A grader that reads the agent's final message rewards agents that say they finished. Checking the system of record removes the incentive.

  • Uncalibrated judges

    Model-based graders drift from human judgement unless their verdicts are compared with human labels on the same transcripts, repeatedly.

How Motiontic's engineers apply it

Motiontic's engineers use code graders wherever the outcome can be checked in the system itself, such as a database row, a passing test or a saved draft, and model graders only for qualities code cannot check. Model graders use rubrics calibrated against labels from the client's own reviewers, and every new grader is checked by reading a sample of the transcripts it passed and failed.

Grader types compared

Grader types compared
Code-basedModel-basedHuman
ExamplesUnit tests, state checks, tool-call checksRubric scoring, pairwise comparisonExpert review, spot checks
Speed and costFast and cheapSlower, costs model callsSlow and expensive
ConsistencyDeterministicNon-deterministicVaries by reviewer
WeaknessBrittle to valid variationsNeeds calibration against humansHard to scale
FitsVerifiable outcomesOpen-ended output, tone, groundednessCalibration and subjective quality

Common questions

What is a grader in an AI eval?

A grader is the part of an eval that scores an attempt at a task. It can be code that checks the result, a model applying a rubric, or a person reviewing the output.

What is the difference between a grader and an LLM judge?

An LLM judge is one kind of grader, the model-based kind. Graders also include deterministic code checks and human reviewers, and many tasks use more than one.

Should an agent grader check the steps or the result?

Anthropic recommends grading the outcome, so the agent is free to find valid paths no one anticipated. Step checks are added where the path is itself a requirement, such as a verification that must come before a payment.

How is a model-based grader calibrated?

By scoring the same transcripts with the model grader and with human reviewers, adjusting the rubric until they agree closely enough, and repeating the comparison as the agent and model change.

Running agents that have to hold up?

Describe the agent, the systems it touches and what it must never get wrong. The first call is with a lab engineer, who will say plainly whether it is ready for production, and whether Motiontic is the right team.