AI Commerce Evals

Automated shopping benchmarks across stores and AI models: multi-turn sequences, cross-model comparison and PDF reports. Built on the same engine as the Agent and the leaderboard.

View sample report Evals and collections docs

Example sequence

multi_item_checkout

Turn 1Show me two different products under $60 each
Turn 2Add both of them to my cart
Turn 3Proceed to checkout

Each turn is a separate API call. The agent sees the whole conversation, with up to 8 tool-calling steps per message.

How it works

1
Define
Pick stores and models and write a multi-turn sequence. Generic prompts work on any catalogue.
2
Run
Every store, model and sequence combination runs as its own multi-turn run: 2 stores and 2 models make 4 runs.
3
Report
A funnel comparison, a performance table, error types and a PDF you can download and share.

Sample results

Four runs of a three-turn sequence on two stores, from the sample report.

Checkout rate: 100%
Runs: 4
Avg tokens: 95,513
Avg duration: 48.3s
Errors: 0

Store	Model	Outcome	Tokens	Duration
oakywood.shop	Gemini 3.5 Flash	Reached checkout	154,294	34.7s
oakywood.shop	Gemini 3.1 Pro	Reached checkout	85,614	93.4s
ugmonk.com	Gemini 3.5 Flash	Reached checkout	96,058	29.9s
ugmonk.com	Gemini 3.1 Pro	Reached checkout	46,084	35.1s

Download the full report

Use cases

For store owners

Run evals on your own store to check that multi-item carts, variants and checkout work the way your UCP profile says.

For agent builders

Compare models head to head on the same stores and find which reaches checkout fastest and cheapest.

For CI/CD

Start eval runs on every deploy and assert on checkout rate, error count and duration, so a regression stops before production.

Headless API

Create a collection, run it, and read the results with a bearer token.

Create

curl -X POST \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  https://ucpplayground.com/api/v1/collections \
  -d '{
    "name": "Q1 Benchmark",
    "config": {
      "stores": ["oakywood.shop", "ugmonk.com"],
      "models": ["gemini-3-flash", "gemini-3-1-pro"],
      "sequences": [{
        "name": "browse_and_buy",
        "turns": [
          { "message": "Show me two products under $60" },
          { "message": "Add both to my cart" },
          { "message": "Proceed to checkout" }
        ]
      }]
    }
  }'

Run

curl -X POST \
  -H "Authorization: Bearer $TOKEN" \
  https://ucpplayground.com/api/v1/collections/{id}/run

Results

curl \
  -H "Authorization: Bearer $TOKEN" \
  https://ucpplayground.com/api/v1/collection-runs/{run_id}

Ready to benchmark your store?

Write a sequence, pick your models, and get a report in under 3 minutes. More in the Evals & Collections help docs.

Create a collection