Understanding Results

Understanding results

When a run ends, its page tells you how far the agent got, why it stopped, whose fault any failure was, and what the run cost. Here is how to read each part.

How it ended

Every run ends at the furthest stage its calls reached, in the same words on the run page, in its Info tab and in History:

  • Searched only: the agent searched but built no cart
  • Built a cart: the store accepted a cart
  • Reached checkout: the store opened a checkout
  • Tried to place the order: the agent sent Complete Checkout and the store did not confirm an order (a declined test card ends here)
  • Order placed: the store confirmed the order
  • Failed: the run broke before it got anywhere

A run counts as reaching checkout only when the store's response shows it: a checkout URL, or a checkout status or order returned by the store. A run through the store's website (WebMCP) is read the same way. It stops at the store's checkout and never pays, so it can't go further than checkout (see WebMCP: the store page).

A run where checkout goes to the store agent (A2A) is read from the store agent's answers, the same checkout object a REST call returns. In an Embedded run the agent never pays: the run reads Order placed when the buyer finishes in the store's checkout, the page sends ec.complete, and the Playground reads the checkout back with get_checkout and the store answers completed with an order (see Embedded: the store's checkout).

The Info tab also says Why a run stopped when there is a reason: the model provider failed, the model refused, it hit the turn limit or the token limit, it stopped on an error, every call failed, it made no progress, or it called no tool.

Stages, step by step

Under the ending sits the funnel, read for the kind of store. A Retail run is read against search, details, cart and checkout; a Stays run against search stays, hold room, book and booked. A confirmed purchase or booking counts as complete even when middle steps were legitimately skipped. The API and the conformance pages use outcome codes for the same endings (search_only, cart_created, checkout_reached, purchase_completed, failed); the full ladder and its variants are in Funnel & Outcomes. A run that built a cart, for example, reads like this:

✓search returned actionable results8 items · ids + prices present
✓product details resolved a variantsize 10 · in stock
✓cart accepted the line itemline item + total present
·checkout completednot reached

Stages are ordered, and the list always shows them all. A · is not a pass: it means the stage was never reached, and the first gap explains everything below it.

Note

Some stores make the details step unnecessary. If search results already carry variant IDs, a capable agent can go from search straight to the cart in fewer turns. The funnel records the steps that actually happened.

Whose fault

When a call fails, the run says whose it was: the store (its endpoint broke or answered wrongly), the agent (it called a tool that doesn't exist, or sent arguments the tool's schema rejects), or ours (our harness: our signing, a key we reused, or our guard holding back a repeated call). The Info tab counts them as store · agent · ours, and each failed line in the Trace is marked. A store's coded answer, such as payment_declined or price_changed, is the store answering, and counts against no one.

Tokens and cost

The Trace header and the Info tab show what the run used:

  • Tokens: in and out. Tokens in covers the prompt, tool definitions and the conversation so far, and grows each turn. Tokens out is what the model wrote: its reasoning, tool calls and replies.
  • Cost: the run's model cost in dollars. Use tokens and cost to compare models on the same task: some reach checkout on a fraction of the tokens others use.

Timing

  • Duration: wall-clock time for the whole run
  • Store time and Model time: how much of that was spent waiting on the store and on the model, shown in the Trace header
  • Per call: every line in the Trace carries its own time, so a slow tool stands out
  • Steps: the number of calls in the trace. The Timeline tab lays them out by stage, by tool, or by tool within each stage.
Warning

A run that hits the turn limit (8 model turns by default) is often looping: retrying a failed call or fetching the same data again. Open the Timeline or the Trace to see where it went round.

If the model provider cuts an answer off part-way, that turn is retried once; a second failure ends the run with an error instead of showing a half-finished answer.

Next steps

If a run didn't end where you expected, read its Trace for the failing call. See Common errors for the most frequent issues and how to resolve them.

Something wrong on this page?