Comparing models
"Which model?" has answers at different levels: the leaderboard for totals across stores, the model pages for head-to-head figures, and your own runs for how models do on your store.
Where to compare
- The Leaderboard: rankings across every run in the Playground. Best for the broad question: which models reliably reach checkout, and at what token cost.
- Models compare: pick up to four models and see their figures side by side: checkout rate, tokens, duration, turns to checkout. Best for a shortlist before you run anything.
- Model pages: one page per model, with its figures and known issues.
- Your own runs: start the same task at the same store on Runs, once per model, changing only the Model pill. Then open them from History (filtered to that store) and compare how each ended, its trace, tokens and cost. Best for seeing how models differ on your store.
Key figures to compare
Wherever you compare, look at three things:
- Checkout rate: the most important figure. A model that reaches checkout 90% of the time handles 9 out of 10 shopping tasks. A few points matter at scale.
- Average tokens: this is cost. Token-efficient models reach checkout on a fraction of the tokens others use for the same task, and at volume the difference is large.
- Average duration: how long the buyer waits. Flash and mini models are generally fastest; large reasoning models are slowest but can handle harder catalogs. The leaderboard's duration column has the measured numbers.
Token efficiency against reliability
There is often a trade-off between tokens and how far a model gets. Smaller, faster models use fewer tokens per run but may fail on stores with complex product schemas, for example products with several option dimensions (color, size, material) that need careful variant choice. Larger models tend to handle these cases better, at a higher token cost.
The leaderboard makes this trade-off visible. Sort by checkout rate to find the most reliable models, then check their token use to see the cost.
Store compatibility
Not every model does equally well on every store. A model that does well on Shopify stores might struggle with a custom UCP implementation that returns data in unexpected shapes. The leaderboard adds up results across stores, so a model's overall score reflects how versatile it is. If you are building for one store or platform, your own runs there will tell you more than the overall ranking.
When you compare models on your own store, use one specific, realistic task, such as "Find a blue cotton t-shirt in size medium under $40", and keep the other pills the same. Same store, same task, same settings: the only difference is the model.
Comparing routes, not models
On a store whose website also registers tools (WebMCP), you can hold the model fixed and change the route instead: run the same task with How the agent reaches the store set to UCP, then to Website, and compare the two runs. See WebMCP: the store page.
Building your own benchmarks
The leaderboard is a starting point, not the final answer. Models change often, store implementations change, and your use case may weigh the figures differently. Run the stores you care about with the tasks your buyers will actually send. For guidance on picking models for different jobs, see Choosing models.
Something wrong on this page?