What are tasks, trials and transcripts in AI evals?
In AI agent evals, a task is one test with defined inputs and success criteria, a trial is one attempt at that task, and a transcript is the full record of what happened during the trial.
Agent evals borrowed words from several fields, and teams often use them loosely. Anthropic's January 2026 guide pins down the vocabulary most of the field now uses: task, trial, grader, transcript, outcome, evaluation harness and agent harness.
The distinctions matter because agents are non-deterministic. One task needs several trials, each trial produces its own transcript, and the score is built from the outcomes of all of them.
The terms
Task
Also called a problem or test case: a single test with defined inputs and success criteria.
Trial
One attempt at a task. Several are run per task because output varies between runs.
Transcript
Also called a trace or trajectory: the full record of a trial, including outputs, tool calls, reasoning and intermediate results. For the Anthropic API it is the complete messages array.
Outcome
The final state of the environment after the trial: whether a reservation exists in the database, not whether the agent claimed to make one.
Grader
The logic that scores some aspect of the trial, by code, model or human review.
Evaluation harness and agent harness
The evaluation harness runs evals end to end. The agent harness, or scaffold, is the system that lets a model act as an agent. Evaluating an agent means evaluating the harness and the model together.
Why trials must be isolated
Each trial should start from a clean environment. Anthropic warns that shared state between trials, such as leftover files, cached data or exhausted resources, causes correlated failures or inflated scores; in one case Claude examined the git history left behind by earlier trials. Trials that all fail because of the same memory limit are not independent, and averaging them produces a misleading number.
The environment is part of the result. Anthropic's February 2026 study found that container resource settings alone moved agentic coding scores by up to 6 percentage points, which is covered under infrastructure noise.
Reading transcripts
Anthropic's guide says eval scores should not be taken at face value until someone has read some transcripts. Transcripts expose graders that pass the wrong thing, tasks that are impossible as written and agents that reach the right outcome by a route no one would accept.
A failure should also be fair: the transcript should show clearly what went wrong and why. If it does not, the task or the grader needs fixing before the score means anything.
How Motiontic's engineers apply it
Motiontic's engineers store eval transcripts in the same format as production traces where possible, so a failure found in production can be replayed as a new task with the same tooling. Trials run in fresh environments with their resource limits recorded alongside the result.
Eval vocabulary at a glance
| Term | What it is | Example |
|---|---|---|
| Task | One test with inputs and success criteria | Refund a duplicate charge for a verified customer |
| Trial | One attempt at the task | Run 3 of 5 for that task |
| Transcript | Full record of the trial | Messages, tool calls and results for run 3 |
| Outcome | Final state of the environment | Exactly one refund row in the payments table |
| Grader | Logic scoring an aspect of the trial | A code check that one refund exists and none were duplicated |
| Evaluation harness | Infrastructure that runs it all | Loads tasks, runs trials, applies graders, reports |
Common questions
What is the difference between a task and a trial?
A task is the test itself: inputs plus success criteria. A trial is one attempt at it. Running five trials of one task shows how consistently the agent succeeds.
Is a transcript the same as a trace?
Largely, yes. Anthropic treats transcript, trace and trajectory as names for the same record. Observability tools usually call it a trace and store it as spans, often in the OpenTelemetry format.
Why run more than one trial per task?
Because agents do not give the same result every run. Several trials give a success rate per task and allow pass@k and pass^k to be calculated.
What is the difference between the outcome and the transcript?
The transcript is what the agent did and said. The outcome is the state it left behind. Grading the outcome avoids trusting an agent's claim that it finished.
Related
Grader (AI evals)
The part of an eval that decides pass or fail.
pass@k and pass^k
Can the agent do it at all, versus does it do it every time.
Eval harness
The machinery that runs evals, separate from the agent it tests.
Infrastructure noise (evals)
Score changes that come from the machine, not the model.
AI agent development
Agents that do real work for users, with the evals, traces and guardrails to keep them in production.
Running agents that have to hold up?
Describe the agent, the systems it touches and what it must never get wrong. The first call is with a lab engineer, who will say plainly whether it is ready for production, and whether Motiontic is the right team.