85 agentic tools audited for security incidents · Talk to the lab →
MotionticBook a call
Glossary

What is trajectory evaluation?

Trajectory evaluation scores the sequence of steps an AI agent took, such as which tools it called, in what order and with which arguments, against a reference path or rubric, rather than scoring only the final answer.

Two agents can reach the same correct answer by very different routes. One checks the customer's account, reads the policy and issues a refund. The other issues the refund straight away and happens to be right. A final-answer eval scores them the same; a trajectory eval does not.

Trajectory evaluation looks at the path: the tools called, their order, their arguments and how many steps it took. It matters most where the process is itself a requirement, and where an inefficient or risky path is a problem even when the outcome is fine.

Common trajectory metrics

  • Exact match

    The agent's tool calls match the reference sequence exactly.

  • In-order match

    All reference calls appear in the right order, with extra calls allowed.

  • Any-order match

    All reference calls appear, in any order.

  • Precision and recall

    How many of the agent's calls were in the reference, and how many reference calls the agent made.

  • Single tool use

    Whether one specific tool was called at all, such as a sanctions check before a payment.

What the labs say

Google's Gemini Enterprise Agent Platform evaluation service ships the metrics above (trajectory_exact_match, trajectory_in_order_match, trajectory_any_order_match, trajectory_precision, trajectory_recall and trajectory_single_tool_use) and evaluates the final response alongside them. Google's ADK adds tool_trajectory_avg_score and a rubric-based, model-graded tool use quality criterion.

OpenAI's agent eval guidance checks tool selection and argument precision for single agents and handoff accuracy for multi-agent systems. Anthropic takes a different line: grade the outcome the agent produced, not the exact path, so the agent is free to find valid solutions no one planned for.

When the path matters

  • Mandatory steps

    Regulated work where a check must happen before an action, whatever the outcome. See the compliance agents page.

  • Cost and latency

    An agent that reaches the right answer in 30 tool calls instead of 4 is a cost problem.

  • Risky side effects

    A path that touched a system it did not need to is worth flagging even when the answer is right.

  • Debugging

    When an outcome eval fails, the trajectory shows which step went wrong.

How Motiontic's engineers apply it

Motiontic's engineers grade the outcome first and add trajectory checks only for steps that are genuinely required, such as an approval or a policy lookup, using single-tool or in-order checks rather than exact matches. Exact-match trajectories penalise valid alternative routes and break every time a tool is renamed, so they are reserved for fixed, regulated sequences.

Outcome grading compared with trajectory evaluation

Outcome grading compared with trajectory evaluation
Outcome gradingTrajectory evaluation
ScoresThe final answer or end stateThe tool calls, order and arguments
NeedsA reference outcome or rubricA reference trajectory or step rubric
StrengthLeaves room for valid, unplanned solutionsCatches skipped checks, waste and risky steps
Main riskMisses a right answer reached the wrong wayPenalises valid alternative paths
FitsMost tasks with a checkable resultRegulated processes, cost control, debugging

Common questions

What is trajectory evaluation?

Trajectory evaluation is scoring the steps an AI agent took, its tool calls, their order and arguments, against a reference path or rubric, as well as or instead of scoring its final answer.

Should agent evals grade the trajectory or the outcome?

Most teams weight the outcome and add trajectory checks for steps that are required. Anthropic recommends outcome grading so agents can find unplanned valid paths; Google and OpenAI also evaluate the path.

What is a reference trajectory?

A reference trajectory is the expected sequence of tool calls for a task, written by someone who knows the process. Agent runs are compared with it using exact, in-order, any-order, precision or recall metrics.

Running agents that have to hold up?

Describe the agent, the systems it touches and what it must never get wrong. The first call is with a lab engineer, who will say plainly whether it is ready for production, and whether Motiontic is the right team.