Reading reports
Every collection run produces a report. Here is how to read the funnel matrix, the performance table and the PDF benchmark report.
Open a collection run from its collection's page. The run page shows the headline numbers (checkout rate, runs, average tokens, average duration, errors) and one row per run: store, model, the four funnel stages, outcome, tokens, duration and a link to the run's replay. Download Report (PDF) gives you the full report.
Funnel comparison matrix
The funnel matrix shows whether each store and model combination got through each shopping stage:
- Search: did the agent call the search tool and get results?
- Details: did the agent fetch product details? (It can show FAIL when the step was skipped but not needed.)
- Cart: did the agent add items to a cart?
- Checkout: did the run reach checkout? It counts only when the store's response shows it: a checkout URL, or a checkout status or order returned by the store.
"Details: FAIL" doesn't always mean something went wrong. Some stores include full variant data in search results, so the agent skips the details step and goes straight to the cart. If Cart and Checkout both pass, the skipped Details step is fine.
Performance table
For each run, the report shows:
- Tokens: total tokens used (prompt and completion). Lower is cheaper.
- Duration: wall-clock time for the whole multi-turn run, including model time and tool call latency.
- Turns: rounds of tool calls across all the sequence's turns. Fewer means a more efficient model.
- Cart value: the cart's total, with its currency, if the agent built a cart.
Recommendations
The PDF report writes recommendations from the results, for example:
- Per store: "example-store.com: checkout rate is 50%. Review MCP tool responses for missing checkout URLs…"
- Token use: "Average token usage is 95,000 — above the 40K baseline…"
- Duration: "Average session duration is 48.0s — above 15s target…"
When every run reaches checkout with no errors, the report says the store is fully AI-agent ready.
PDF report
Every collection run can be downloaded as a PDF, the "AI Commerce Benchmark Report". It contains:
- A header with the date, UCP version, models, stores and number of runs
- Headline numbers: checkout rate, runs, average tokens, average duration, errors
- The funnel comparison matrix
- The performance table, with totals
- A breakdown by store
- The eval's set-up (stores, models, sequence turns)
- Recommendations
- Links to each run's replay
The PDF is made to be shared: send it to your developers, add it to a client pitch, or attach it to a ticket. Each run ID in the report links to that run's replay.
Comparing collection runs
Run the same collection every week to track progress. Each collection has a version: change its stores, models or sequences and the version goes up, and each collection run shows the version it used. Compare one week with the next to see whether your store's checkout rate is improving or slipping.
Something wrong on this page?