What Are Evals?

Evals are automated shopping benchmarks that test how AI models perform against real stores. Define a multi-turn conversation, pick your stores and models, and get a structured report comparing every combination.

The Problem Evals Solve

Running a single agent session tells you what happened once. But shopping agent behavior varies -- the same model can succeed on one store and fail on another, or succeed with one prompt and stall with a slightly different one. You need repeatable, structured testing to draw real conclusions.

Evals let you define a test once and run it across any combination of stores and models. Every session uses the same sequence of messages, so results are directly comparable.

How They Work

A collection is a saved eval configuration containing:

• Stores -- one or more domains to test against (e.g., oakywood.shop, ugmonk.com)
• Models -- one or more AI models to compare (e.g., Gemini 3.5 Flash, Gemini 3.1 Pro)
• Sequences -- multi-turn conversation scripts that simulate a shopping session

Running a collection executes every combination. For example: 2 stores x 2 models x 1 sequence = 4 sessions, each following the same scripted conversation.

Multi-Turn Sequences

Each sequence is a series of user messages sent one at a time. The agent responds to each, and the next message is sent as a follow-up -- just like a real conversation.

A typical 3-turn sequence:

1. T1: "Show me two different products under $60 each"
2. T2: "Add both of them to my cart"
3. T3: "Proceed to checkout"

Each turn gets its own orchestrator round (up to 8 internal tool-calling turns), and the full conversation history carries forward between turns. This is more reliable than cramming everything into a single prompt.

What You Get

After a run completes, you get:

• Funnel comparison matrix -- which store+model combinations passed search, details, cart, and checkout
• Performance table -- tokens, duration, turns, and cart value for each session
• Error classification -- tool errors, schema failures, and model refusals categorized by who needs to fix them
• Recommendations -- auto-generated actionable suggestions based on the results
• PDF report -- downloadable benchmark report ready to share with your team or clients
• Session replays -- every session is replayable with the full tool call timeline