What are evals?

Evals are repeatable shopping tests. You write a conversation once, pick the stores and models, and get one report that compares every combination.

The problem evals solve

One run tells you what happened once. Agent behaviour varies: the same model can succeed on one store and fail on another, or succeed with one prompt and stall with a slightly different one. To draw conclusions you need the same test, run the same way, many times.

An eval runs one test across any mix of stores and models. Every run sends the same messages, so the results compare directly.

How they work

A collection is a saved eval. It holds:

• Stores: one or more domains to test (for example oakywood.shop, ugmonk.com)
• Models: one or more models to compare (for example Gemini 3.5 Flash, Gemini 3.1 Pro)
• Sequences: scripted conversations, one buyer message per turn

Running a collection starts one run for every combination. For example, 2 stores × 2 models × 1 sequence = 4 runs, each following the same script.

Multi-turn sequences

A sequence is a list of buyer messages sent one at a time. The agent answers each one, then the next message goes in as a follow-up, as in a real conversation.

A typical three-turn sequence:

1. T1: "Show me two different products under $60 each"
2. T2: "Add both of them to my cart"
3. T3: "Proceed to checkout"

Each turn gets up to 8 rounds of tool calls, and the whole conversation carries forward from turn to turn. This works more reliably than putting everything into one message.

What you get

When a collection run finishes, you get:

• Funnel comparison matrix: which store and model combinations got to search, details, cart and checkout
• Performance table: tokens, duration, turns and cart value for each run
• Error count: tool errors across all the runs
• Recommendations: suggestions written from the results
• PDF report: a benchmark report you can share with your team or clients
• Run replays: every run can be replayed with its full trace of tool calls

See Reading reports for each part.