AI Commerce Evals

Automated shopping benchmarks across stores and AI models. Multi-turn sequences, cross-model comparison, and PDF reports. Built on the same engine that powers the agent benchmark and leaderboard.

View Sample Report

How It Works

Define

Pick stores, models, and write a multi-turn sequence. Generic prompts work across any catalog.

Run

Every store x model x sequence combination runs as an independent multi-turn session. 2 stores x 2 models = 4 sessions.

Report

Get a funnel comparison matrix, performance table, error classification, and a downloadable PDF report.

Example Sequence

multi_item_checkout

T1Show me two different products under $60 each

T2Add both of them to my cart

T3Proceed to checkout

Each turn is a separate API call. The agent sees full conversation history. Up to 8 internal tool-calling turns per message.

Sample Results

100%Checkout Rate

4Sessions

95,513Avg Tokens

48.3sAvg Duration

0Errors

Store	Model	Outcome	Tokens	Duration
oakywood.shop	Gemini 3.5 Flash	checkout reached	154,294	34.7s
oakywood.shop	Gemini 3.1 Pro	checkout reached	85,614	93.4s
ugmonk.com	Gemini 3.5 Flash	checkout reached	96,058	29.9s
ugmonk.com	Gemini 3.1 Pro	checkout reached	46,084	35.1s

Download full report →

Use Cases

For Store Owners

Run evals against your own store to verify your UCP implementation handles multi-item carts, variants, and checkout flows correctly.

For Agent Builders

Compare models head-to-head on the same stores. Find which model reaches checkout fastest and cheapest.

For CI/CD

Trigger eval runs on every deploy. Assert on checkout rate, error count, and duration thresholds. Catch regressions before production.

Headless API

Create

curl -X POST -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  https://ucpplayground.com/api/v1/collections \
  -d '{
    "name": "Q1 Benchmark",
    "config": {
      "stores": ["oakywood.shop", "ugmonk.com"],
      "models": ["gemini-3-flash", "gemini-3-1-pro"],
      "sequences": [{
        "name": "browse_and_buy",
        "turns": [
          { "message": "Show me two products under $60" },
          { "message": "Add both to my cart" },
          { "message": "Proceed to checkout" }
        ]
      }]
    }
  }'

Run

curl -X POST -H "Authorization: Bearer $TOKEN" \
  https://ucpplayground.com/api/v1/collections/{id}/run

Results

curl -H "Authorization: Bearer $TOKEN" \
  https://ucpplayground.com/api/v1/collection-runs/{run_id}

Learn more in the Evals & Collections help docs.