How Scoring Works

How scoring works

The leaderboard and the model pages rank models by how well they finish real shopping tasks: runs against UCP stores, not synthetic benchmarks.

The main measure: checkout rate

The checkout rate is the share of a model's runs whose outcome is checkout_reached or purchase_completed (a confirmed purchase or booking has reached checkout too). A model that gets there in 8 of 10 runs scores 80%. To get there, a model has to search well, pick the right product, handle variants, build a cart and get through checkout without getting stuck or looping.

Two more rates sit beside it, each counting every run that got at least that far:

  • Cart rate: runs that built a cart, reached checkout or paid.
  • Search rate: runs that searched, built a cart, reached checkout or paid.

The leaderboard's views

ViewRanked byShows
FunnelCheckout rate, highest first Search, cart and checkout rates; average tokens and duration per run.
EfficiencyFewest turns to checkout Average turns, tokens and duration on the runs that reached checkout. Only models with at least 3 runs appear.
TokensFewest tokens per runAverage tokens per run, across all runs.
  • Tokens: how many tokens the model uses. Fewer tokens means a lower cost per run.
  • Duration: wall-clock time for the run, which reflects both the model's speed and how many round trips it needs.
  • Turns: rounds of tool calls in a run. Fewer turns generally means the model chooses better which tools to call, and in what order.

The shopping score

The model pages give each model a shopping score from 0 to 100, made of five parts:

PartWeightHow it is measured
Checkout rate40%As above.
Cart rate20%As above.
Search rate10%As above.
Turn efficiency15% Average turns on runs that reached checkout: 4 turns or fewer scores 100, 12 or more scores 0, in a straight line between.
Token efficiency15% Average tokens on runs that reached checkout: 20,000 or fewer scores 100, 100,000 or more scores 0, in a straight line between.

A model with no run that reached checkout gets 50 for each efficiency part. The score is rounded to a whole number and shown with a word:

ScoreWord
75–100Strong
60–74Good
35–59Fair
0–34Weak

The models list shows models with at least 2 runs, sorted by score unless you choose Checkout or Name. Each model's page breaks its score into these five parts.

Where the data comes from

The numbers come from every run on the Playground: yours, other people's, runs by visitors who aren't signed in, and eval runs alike. When anyone picks a model, gives it a shopping task and lets it work against a store, that run counts. There is no separate hand-picked benchmark set, and no simulated responses or scripted stores. Two kinds of run are left out: runs through a store's website (WebMCP), and runs on your own runner (the Local runtime). Their results stay with the person who ran them. Only models in the current model list are ranked.

So the scores reflect real model behaviour: odd product schemas, recovering from tool errors, and the variety of real store data. They update as new runs finish.

Note

Scores move when stores change their UCP set-up or when model providers release new versions. A model's place today may differ from last week's if the model behind it has changed.

Understanding the funnel

The checkout rate measures the last stage of the shopping funnel. To see how runs break down across each stage, from search through cart to checkout, read Funnel & outcomes.

Something wrong on this page?