What is an eval harness?
An eval harness is the infrastructure that runs an evaluation end to end: it loads tasks, runs the agent or model on each one, records transcripts, applies graders and aggregates the scores.
Checked
Writing eval tasks is half the work. Something has to run them: start a clean environment for each trial, run the agent, capture everything it did, apply each grader and turn the results into a score a team can compare across versions. That machinery is the eval harness.
Anthropic's agent eval guide separates it from the agent harness, the system that lets a model act as an agent. One is the thing under test; the other is the test rig.
What an eval harness does
Loads tasks
Reads the dataset, with inputs, references and success criteria, at a known version.
Isolates each trial
Starts each attempt in a fresh environment, often a container, so one run cannot affect the next.
Runs trials
Executes each task several times, in parallel where possible, with fixed resource limits.
Records transcripts
Keeps the full log of every trial for reading and debugging.
Applies graders
Runs code checks, LLM judges and human review queues on each trial.
Reports and gates
Aggregates scores per task and slice, compares them with a baseline and fails the build when a threshold is missed.
Open options in 2026
Inspect, from the UK AI Security Institute and Meridian Labs, is an MIT-licensed framework with sandboxes including Docker and Kubernetes and more than 200 pre-built evals. Promptfoo is an MIT-licensed CLI for evals and red-teaming in CI; OpenAI announced it would acquire Promptfoo in March 2026 and names it as the migration path from OpenAI's Evals platform, which shuts down on 2026-11-30. Harbor runs containerised agent evals and is listed in Anthropic's guide. DeepEval, MLflow and Google's ADK eval tooling are further options.
Whatever the tool, Anthropic's research on infrastructure noise is a reason to record the harness configuration with every score: container limits alone moved agentic coding results by up to 6 points.
How Motiontic's engineers apply it
Motiontic's engineers keep the eval harness in the client's repository, next to the agent, and run it in the client's CI. The harness is chosen to fit the client's stack rather than a house default, and is documented so the client's team can add tasks and graders without outside help.
Eval harness compared with agent harness
| Eval harness | Agent harness | |
|---|---|---|
| Role | Runs tests against the agent | Lets the model act as an agent |
| Contains | Tasks, trial runner, graders, reporting | Agent loop, tools, context, permissions, state |
| Runs | In development and CI | In production, for every user request |
| Output | Scores and pass rates | Actions and answers |
Common questions
What is an eval harness?
An eval harness is the software that runs evals: it loads tasks, runs the system on each one in isolation, records transcripts, applies graders and reports scores.
What is the difference between an eval harness and an agent harness?
The agent harness is the loop, tools and state that let a model act as an agent. The eval harness is the separate infrastructure that runs tasks against that agent and scores the results.
What replaces the OpenAI Evals platform?
OpenAI's deprecations page names Promptfoo as the migration path. The hosted Evals platform went read-only on 2026-10-31 and shuts down on 2026-11-30.
Related
Agent harness
The loop, tools, context, permissions and state that sit around the model.
Eval
Repeatable, scored tests that tell a team whether an AI change made things better or worse.
Regression eval
The suite that catches what a change broke before users do.
Harness engineering
Designing the environment an agent works in, so its output can be checked and shipped.
AI agent development
Agents that do real work for users, with the evals, traces and guardrails to keep them in production.
Running agents that have to hold up?
Describe the agent, the systems it touches and what it must never get wrong. The first call is with a lab engineer, who will say plainly whether it is ready for production, and whether Motiontic is the right team.