85 agentic tools audited for security incidents · Talk to the lab →
MotionticBook a call
Glossary

What are tasks, trials and transcripts in AI evals?

In AI agent evals, a task is one test with defined inputs and success criteria, a trial is one attempt at that task, and a transcript is the full record of what happened during the trial.

Agent evals borrowed words from several fields, and teams often use them loosely. Anthropic's January 2026 guide pins down the vocabulary most of the field now uses: task, trial, grader, transcript, outcome, evaluation harness and agent harness.

The distinctions matter because agents are non-deterministic. One task needs several trials, each trial produces its own transcript, and the score is built from the outcomes of all of them.

The terms

  • Task

    Also called a problem or test case: a single test with defined inputs and success criteria.

  • Trial

    One attempt at a task. Several are run per task because output varies between runs.

  • Transcript

    Also called a trace or trajectory: the full record of a trial, including outputs, tool calls, reasoning and intermediate results. For the Anthropic API it is the complete messages array.

  • Outcome

    The final state of the environment after the trial: whether a reservation exists in the database, not whether the agent claimed to make one.

  • Grader

    The logic that scores some aspect of the trial, by code, model or human review.

  • Evaluation harness and agent harness

    The evaluation harness runs evals end to end. The agent harness, or scaffold, is the system that lets a model act as an agent. Evaluating an agent means evaluating the harness and the model together.

Why trials must be isolated

Each trial should start from a clean environment. Anthropic warns that shared state between trials, such as leftover files, cached data or exhausted resources, causes correlated failures or inflated scores; in one case Claude examined the git history left behind by earlier trials. Trials that all fail because of the same memory limit are not independent, and averaging them produces a misleading number.

The environment is part of the result. Anthropic's February 2026 study found that container resource settings alone moved agentic coding scores by up to 6 percentage points, which is covered under infrastructure noise.

Reading transcripts

Anthropic's guide says eval scores should not be taken at face value until someone has read some transcripts. Transcripts expose graders that pass the wrong thing, tasks that are impossible as written and agents that reach the right outcome by a route no one would accept.

A failure should also be fair: the transcript should show clearly what went wrong and why. If it does not, the task or the grader needs fixing before the score means anything.

How Motiontic's engineers apply it

Motiontic's engineers store eval transcripts in the same format as production traces where possible, so a failure found in production can be replayed as a new task with the same tooling. Trials run in fresh environments with their resource limits recorded alongside the result.

Eval vocabulary at a glance

Eval vocabulary at a glance
TermWhat it isExample
TaskOne test with inputs and success criteriaRefund a duplicate charge for a verified customer
TrialOne attempt at the taskRun 3 of 5 for that task
TranscriptFull record of the trialMessages, tool calls and results for run 3
OutcomeFinal state of the environmentExactly one refund row in the payments table
GraderLogic scoring an aspect of the trialA code check that one refund exists and none were duplicated
Evaluation harnessInfrastructure that runs it allLoads tasks, runs trials, applies graders, reports

Common questions

What is the difference between a task and a trial?

A task is the test itself: inputs plus success criteria. A trial is one attempt at it. Running five trials of one task shows how consistently the agent succeeds.

Is a transcript the same as a trace?

Largely, yes. Anthropic treats transcript, trace and trajectory as names for the same record. Observability tools usually call it a trace and store it as spans, often in the OpenTelemetry format.

Why run more than one trial per task?

Because agents do not give the same result every run. Several trials give a success rate per task and allow pass@k and pass^k to be calculated.

What is the difference between the outcome and the transcript?

The transcript is what the agent did and said. The outcome is the state it left behind. Grading the outcome avoids trusting an agent's claim that it finished.

Running agents that have to hold up?

Describe the agent, the systems it touches and what it must never get wrong. The first call is with a lab engineer, who will say plainly whether it is ready for production, and whether Motiontic is the right team.