Trajectory vs outcome: why agents pass evals and still fail

Outcome evaluation checks whether an AI agent reached the right final result. Trajectory evaluation checks the path it took to get there: which tools it called, in what order, with what arguments, and how many steps it wasted. You need both, because an agent can produce the correct answer through a lucky, fragile path that breaks the moment the input shifts, and it can also take every sensible step and still miss the goal. Use outcome evals to gate releases and trajectory evals to explain failures and catch fragility before it reaches customers.

Most teams ship outcome evals first, because they map to a promise a customer can feel: did the support agent resolve the ticket, did the research agent return the right figure. That is the correct place to start. But once an agent is doing real work over several tool calls, an outcome-only suite tells you that something broke without telling you where, and it quietly passes runs that only worked by accident.

What outcome evaluation misses

An outcome eval collapses an entire run into a single verdict: pass or fail against the final answer. That is cheap to compute and easy to reason about, which is exactly why it should gate your releases. The problem is everything it throws away.

Consider a billing-support agent that answers "why did my invoice go up." It should look up the customer's plan, pull the last two invoices, diff the line items, and explain the delta. Suppose on one test case it skips the plan lookup, guesses the tier from the invoice total, and still lands on the right explanation because the guess happened to be correct. The outcome eval scores it a pass. You ship. Two weeks later a customer on an annual plan with a mid-cycle upgrade hits the same path, the total-based guess is wrong, and the agent confidently explains a delta that never happened.

The run that passed and the run that failed took the same broken path. Outcome evaluation could not distinguish them because it only ever saw the answer. This is the same class of problem as silent failures where a wrong-but-plausible result flows downstream unchecked; the difference is that here the failure is baked into how the agent reasons, not into a single bad tool response.

What a trajectory actually is

A trajectory is the ordered record of everything the agent did between the request and the response: each model turn, each tool call with its arguments, each tool result, each retry, and any handoff to another agent. If you already run tracing you are most of the way there. The gap between observability and evals is that observability captures the trajectory and lets a human read it, while trajectory evaluation scores it automatically so you can catch regressions without staring at traces all day.

To evaluate a trajectory you need it in a structured shape, not a wall of logs: a list of steps where each records the tool name, the arguments the model passed, the raw result, and a timestamp. That structure is what lets you write assertions like "the agent called get_plan before get_invoices" or "no tool was called more than twice" instead of grepping text.

What to score in a trajectory

Trajectory evaluation is not one metric. It is a small set of checks, and most of them are deterministic code, not a judge model. Reach for an LLM as a judge only for the steps that genuinely need semantic reasoning, and calibrate it before you trust it.

Tool selection

Did the agent call the tools the task required, and did it avoid tools it had no business calling? For the billing agent, a run that never calls get_plan is suspect regardless of the final answer. Tool selection is usually a reference-based check: for a labeled test case you know which tools should appear, so you assert on their presence and flag unexpected ones.

Step efficiency

Count the steps and look for waste: the same tool called with identical arguments twice, a search repeated because the first result was ignored, a loop that only exits on a turn limit. Wasted steps are not just a cost line, though they do show up on your bill. They are the earliest signal that an agent is thrashing, and they predict the timeouts and handoff loops between agents that turn into incidents at scale.

Order and dependencies

Some steps only make sense in sequence. Diffing invoices before fetching them is nonsense; issuing a refund before confirming the charge is dangerous. Encode the real dependencies as ordering assertions. These catch the fragile "right answer, wrong order" runs that outcome evals wave through.

Argument correctness

A tool can be the right tool, called at the right time, with the wrong arguments: the correct customer id but last month's date range, or a refund amount off by a currency-conversion factor. Validate the arguments the model constructed, not just the fact that the call happened. This is where trajectory evals overlap with designing agent tools as API products with strict input contracts, because a tool that validates its own inputs turns a silent wrong-argument bug into a loud, catchable one.

Recovery

When a tool returns an error or an empty result, what does the agent do next? A healthy trajectory shows it noticing, adjusting, and either recovering or failing closed; an unhealthy one shows it inventing a value and marching on. Score whether error steps are followed by a sensible response rather than a fabrication.

The two failure modes only trajectory evals catch

Everything above exists to surface two runs that outcome evaluation cannot tell apart from a clean success.

The first is the right answer through the wrong path. The agent lands on the correct output but did it by guessing, by skipping a required check, or by relying on a coincidence in the test data. It passes every outcome eval and is one input change away from being wrong. Trajectory evaluation is the only automated way to find these before a customer does, because the only evidence of the problem is in the path.

The second is the wrong answer through the right path. The agent did everything correctly, and the task still failed because a tool returned stale data, a downstream service was degraded, or the task was genuinely impossible. Outcome evals will blame the agent. Trajectory evaluation shows the steps were sound, which points you at the real cause instead of sending you to rewrite a prompt that was never the problem.

How to combine the two without drowning

The pragmatic split, and the one that keeps your suite fast, is to let outcome evals gate and trajectory evals explain. Outcome evals run on every change and decide whether a build ships. They are binary, cheap, and as deterministic as you can make them, which is why they belong in CI as your release gate. Trajectory evals run alongside them but are advisory: they annotate each failing run with which step went wrong, and they run on a sample of passing runs to catch the fragile-but-lucky cases. A trajectory check firing on a run whose outcome passed is your highest-value signal, because it is a bug you would otherwise have shipped.

The other half of the discipline is feeding what trajectory evals find back into the suite. Every fragile path a trajectory check catches becomes a new labeled case, the same way you should grow your eval set from real production failures. Over a few months your trajectory checks stop being generic and start encoding the specific ways your agent tends to go wrong.

A worked example

Take the billing agent from earlier, running in a B2B SaaS product with a support copilot. At launch it had 30 outcome test cases, all green, and shipped. Numbers here are illustrative, but the shape is real.

In the first month, two escalations came in on inputs the outcome suite had rated as passing. Reading the traces by hand showed the same pattern: on customers with any plan change mid-cycle, the agent skipped get_plan and inferred the tier from the invoice total. On clean monthly accounts the inference was right, so 27 of 30 cases passed and the fragility stayed hidden.

The fix was not a new prompt. It was three trajectory assertions added to the same test cases: get_plan must appear before any invoice diff, no tool may run on an inferred tier when a lookup is available, and any get_invoices error must be followed by a clarifying question rather than an explanation. Re-running the suite, 6 of the 30 cases now failed on trajectory even though their outcomes still passed. Those 6 were the fragile runs. After correcting the tool-calling logic the checks went green, the escalations stopped, and the same assertions later blocked a refactor from silently reintroducing the skip. The outcome number never moved; the trajectory number is what told the team the agent was sound.

When trajectory evaluation is overkill

If your feature is a single model call with no tools, there is no trajectory to evaluate, and outcome evals plus good acceptance criteria written before you build the feature are the whole job. The same is true for a two-step chain simple enough that a wrong path and a wrong answer are effectively the same event. Trajectory evaluation earns its keep once an agent makes enough independent decisions that it can be right by accident or wrong for reasons unrelated to its reasoning. The line moves as the agent accretes tools; watch for the first time you debug a failure by reading the trace instead of the answer, because that is the moment the path became the product.

FAQ

Do trajectory evals replace outcome evals?

No. Outcome evals stay as your release gate because they map to what the customer cares about and are cheap to run. Trajectory evals sit next to them to explain failures and catch runs that pass by luck. If you can only build one, build outcome evals; to ship with confidence month over month, you need both.

Do I need a judge model to evaluate a trajectory?

Mostly no. Tool presence, call counts, ordering, and argument validation are deterministic assertions in plain code, which makes them fast and stable. Save a judge model for the few checks that need semantic reasoning, such as whether a recovery step was sensible, and calibrate that judge against human labels before you rely on it.

How do I get the trajectory data?

From your traces. If you already emit structured spans for each model turn and tool call, you have the raw material; you just need it shaped as an ordered list of steps with tool name, arguments, and result so you can assert on it. Teams that treat tracing as a first-class part of the build, described in our AI engineering playbook, get trajectory evaluation almost for free.