85 agentic tools audited for security incidents · Talk to the lab →
MotionticBook a call
Glossary

What is an eval?

An eval is a repeatable test of an AI system: a set of tasks is run, graders score each attempt, and the results show whether the system performs well enough to ship.

Traditional software tests assume the same input gives the same output. AI systems do not work that way. The same prompt can produce different answers, and many correct answers can look different from each other. Evals are how teams measure quality anyway: run many tasks, often many times, and score the results.

Without evals, every prompt edit or model upgrade is judged by a few spot checks. With them, a team can see whether a change helped, hurt or did nothing, before users find out.

The core vocabulary

  • Task

    One test case with inputs and success criteria.

  • Trial

    One attempt at a task. Agents are often run several times per task because output varies.

  • Grader

    The logic that scores some aspect of a trial: code, an LLM judge or a person.

  • Transcript

    The full record of a trial, including tool calls and intermediate steps.

  • Outcome

    The final state the agent produced, such as a database row, a file or a sent message.

Kinds of eval

Capability evals measure what the system cannot yet do reliably and start with a low pass rate. Regression evals guard what already works and should sit near 100%. Anthropic describes tasks graduating from the first suite to the second as they are solved.

Two reliability measures matter for agents: pass@k asks whether at least one of k attempts succeeds, and pass^k asks whether all k succeed. Offline evals run curated datasets in development and CI; online evals score live production traces with the same graders.

How Motiontic's engineers apply it

Motiontic's engineers build the first eval set from the client's own data, starting with 20 to 50 tasks drawn from real cases and real failures, as Anthropic's guide recommends. Outcomes are graded rather than exact paths, so the agent is free to find a valid route the team did not anticipate.

The suite runs on every prompt, tool or model change, and production failures are turned into new tasks so the same mistake is caught next time.

An eval compared with a software test

An eval compared with a software test
Software testEval
OutputDeterministicVaries between runs
ResultPass or failA score or pass rate across many tasks and trials
GradingExact assertionsCode checks, LLM judges and human review
Target100% passingRegression suites near 100%; capability suites climb over time
When it runsOn every commitOn every prompt, tool, data or model change

Common questions

What is an eval in AI?

An eval is a structured, repeatable test of an AI model or agent. It runs a set of tasks, scores each attempt with code, a model-based grader or a person, and reports how well the system performs.

How many tasks does a first eval set need?

Anthropic's guidance suggests starting with 20 to 50 simple tasks drawn from real failures, reading the transcripts, and growing the set as new failures appear.

What is the difference between pass@k and pass^k?

pass@k measures whether at least one of k attempts succeeds, which shows what the system can do. pass^k measures whether all k attempts succeed, which shows how reliably it does it.

Should evals grade the outcome or the steps?

Labs differ. Anthropic recommends grading the outcome so agents can find valid paths no one planned for; Google and OpenAI also grade the trajectory or trace. Many teams do both, weighting the outcome.

Running agents that have to hold up?

Describe the agent, the systems it touches and what it must never get wrong. The first call is with a lab engineer, who will say plainly whether it is ready for production, and whether Motiontic is the right team.