Creating collections
A collection says what to test: which stores, which models, and which conversations to run. Once it is saved you can run it any time: from its page, through the API, or on a schedule.
Create one at Collections → Create Collection: give it a name, list the stores, pick the models, and write the sequence. The form shows how many runs each collection run will start, and how many your monthly allowance has left.
Choosing stores
List one or more store domains, one per line. Any store whose UCP profile declares an MCP endpoint works: Shopify stores, UCPReady (WooCommerce) stores, or your own UCP build. The eval finds the MCP endpoint from the store's UCP profile at /.well-known/ucp.
Evals reach stores over UCP's MCP transport only. A store that declares REST but no MCP endpoint can't be part of an eval; test it with a run on Runs or through the API instead. Runs through a store's website (WebMCP) are not part of evals.
Start with stores you have already tried with a run of your own. You know their tools work and what to expect. Add new stores once your sequences are proven.
Choosing models
Pick the models to compare. Every model on the Model pill is available. Common patterns:
- Head to head: two models on the same stores (for example Gemini 3.5 Flash against Gemini 3.1 Pro)
- Frontier sweep: every frontier model, to find the best one for your use
- Cost check: a frontier model against cheaper ones, to find the best price for the results
Writing sequences
A sequence is the script an eval runs: a name and a list of turns, each one a buyer message. The form starts you from a preset (Browse & Buy, Multi-Item, Price Constrained) or a blank Custom one. Keep messages general so they work on any store:
Good sequences
- "What do you sell?": works on any store
- "Show me products under $60": works with any catalog
- "Add the first one to my cart": refers to the agent's previous answer
- "Proceed to checkout": works everywhere
Sequences to avoid
- "Find the Nike Air Max 90 in size 10": too specific, works on one store only
- "Search, add to cart, checkout, and pay": too much for one turn, better as three turns
How many runs
Each collection run starts stores × models × sequences runs. Each one is its own multi-turn conversation.
- 2 stores × 2 models × 1 sequence = 4 runs
- 3 stores × 3 models × 2 sequences = 18 runs
- 5 stores × 5 models × 1 sequence = 25 runs
Every run uses model tokens and counts towards your monthly allowance. A collection run of 25 runs costs about the same as 25 runs started by hand. Plan for it before you run a large matrix.
API example
You can also create a collection through the API, with as many sequences as you need:
curl -X POST -H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
https://ucpplayground.com/api/v1/collections \
-d '{
"name": "Q1 Benchmark",
"config": {
"stores": ["oakywood.shop", "ugmonk.com"],
"models": ["gemini-3-flash", "gemini-3-1-pro"],
"sequences": [{
"name": "browse_and_buy",
"turns": [
{ "message": "Show me two products under $60" },
{ "message": "Add both to my cart" },
{ "message": "Proceed to checkout" }
]
}]
}
}'Something wrong on this page?