What Are Evals?
Evals are automated shopping benchmarks that test how AI models perform against real stores. Define a multi-turn conversation, pick your stores and models, and get a structured report comparing every combination.
The Problem Evals Solve
Running a single agent session tells you what happened once. But shopping agent behavior varies -- the same model can succeed on one store and fail on another, or succeed with one prompt and stall with a slightly different one. You need repeatable, structured testing to draw real conclusions.
Evals let you define a test once and run it across any combination of stores and models. Every session uses the same sequence of messages, so results are directly comparable.
How They Work
A collection is a saved eval configuration containing:
- Stores -- one or more domains to test against (e.g., oakywood.shop, ugmonk.com)
- Models -- one or more AI models to compare (e.g., Gemini 3.5 Flash, Gemini 3.1 Pro)
- Sequences -- multi-turn conversation scripts that simulate a shopping session
Running a collection executes every combination. For example: 2 stores x 2 models x 1 sequence = 4 sessions, each following the same scripted conversation.
Multi-Turn Sequences
Each sequence is a series of user messages sent one at a time. The agent responds to each, and the next message is sent as a follow-up -- just like a real conversation.
A typical 3-turn sequence:
- T1: "Show me two different products under $60 each"
- T2: "Add both of them to my cart"
- T3: "Proceed to checkout"
Each turn gets its own orchestrator round (up to 8 internal tool-calling turns), and the full conversation history carries forward between turns. This is more reliable than cramming everything into a single prompt.
Evals produce the same session data as the web UI and headless API. Session replay, leaderboard data, and team sharing all work automatically with eval results. Eval sessions connect over UCP's MCP transport; store-page (WebMCP) runs are Agent-page only.
What You Get
After a run completes, you get:
- Funnel comparison matrix -- which store+model combinations passed search, details, cart, and checkout
- Performance table -- tokens, duration, turns, and cart value for each session
- Error classification -- tool errors, schema failures, and model refusals categorized by who needs to fix them
- Recommendations -- auto-generated actionable suggestions based on the results
- PDF report -- downloadable benchmark report ready to share with your team or clients
- Session replays -- every session is replayable with the full tool call timeline