CI/CD integration
Run an eval on every deploy to catch regressions before they reach production. Call the API from GitHub Actions, GitLab CI or any other CI system.
How it works
- Create a collection with your stores, models and sequences (once)
- Start a collection run with
POST /api/v1/collections/{id}/run - Poll until it finishes with
GET /api/v1/collection-runs/{run_id} - Check the results: assert on checkout rate, error count or duration
GitHub Actions example
name: UCP Store Evals
on:
deployment_status:
types: [completed]
jobs:
eval:
runs-on: ubuntu-latest
steps:
- name: Run collection
run: |
RUN_ID=$(curl -s -X POST \
-H "Authorization: Bearer ${{ secrets.UCP_TOKEN }}" \
https://ucpplayground.com/api/v1/collections/${{ vars.COLLECTION_ID }}/run \
| jq -r '.run_id')
# Poll until complete (max 5 minutes)
for i in $(seq 1 30); do
STATUS=$(curl -s \
-H "Authorization: Bearer ${{ secrets.UCP_TOKEN }}" \
https://ucpplayground.com/api/v1/collection-runs/$RUN_ID \
| jq -r '.status')
[ "$STATUS" = "complete" ] && break
sleep 10
done
# Check checkout rate
RATE=$(curl -s \
-H "Authorization: Bearer ${{ secrets.UCP_TOKEN }}" \
https://ucpplayground.com/api/v1/collection-runs/$RUN_ID \
| jq '.summary.checkout_rate')
echo "Checkout rate: $RATE%"
if (( $(echo "$RATE < 80" | bc -l) )); then
echo "::error::Checkout rate $RATE% below 80% threshold"
exit 1
fiAPI endpoints
| Endpoint | Description |
|---|---|
POST /api/v1/collections | Create a collection |
GET /api/v1/collections | List collections |
POST /api/v1/collections/{id}/run | Start a collection run |
GET /api/v1/collection-runs/{id} | Get a collection run's status and summary |
GET /api/v1/collection-runs/{id}/pdf | Download the PDF report |
Keep your collection ID as a CI variable and your API token as a secret. The collection stays between deploys: create it once, then start a collection run on each deploy.
Scheduled runs
To keep checking for regressions, set a cron schedule on the collection. Collection runs then start on their own while is_active is true:
curl -X PATCH -H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
https://ucpplayground.com/api/v1/collections/{id} \
-d '{ "schedule": "0 9 * * 1" }' # Every Monday at 9am The API rejects invalid cron expressions. List past collection runs with GET /api/v1/collections/{id}/runs to follow the trend. If you would rather start them from your own CI, a scheduled GitHub Actions workflow calling POST /api/v1/collections/{id}/run works just as well.
Completion webhooks
Set a webhook_url (HTTPS only) on the collection to get a POST when each collection run finishes, with no polling:
curl -X PATCH -H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
https://ucpplayground.com/api/v1/collections/{id} \
-d '{ "webhook_url": "https://ci.example.com/hooks/ucp" }' The response includes a webhook_secret. It is shown once, and changes whenever the URL changes. Each delivery is signed with X-Playground-Signature: sha256=<HMAC-SHA256 of the raw body>; check it before you trust the payload. The body carries event: collection_run.completed, the collection run's id and status, and its full summary (checkout rates, breakdowns by store and by model, errors). A failed delivery is retried twice, with a wait between tries.
Assertions
Set thresholds in the collection's config and we check them when each collection run finishes. The summary carries passed_assertions (the webhook payload has it too), so CI can gate on one true-or-false value:
"config": {
"assertions": { "min_checkout_rate": 80, "max_avg_tokens": 50000, "max_errors": 0 }
} Supported: min_checkout_rate, min_search_rate, max_avg_tokens, max_avg_duration_ms, max_errors. The API rejects any other key. Each assertion's threshold and actual value are in summary.assertions.
What to assert on
- checkout_rate >= 80: a store ready for production should reach at least 80%
- errors.total == 0: no tool errors means the MCP side is clean
- avg_duration_ms < 30000: runs over 30 seconds can point to a slow MCP endpoint
- avg_tokens < 50000: high token use points to long product descriptions or an agent going round in circles
Something wrong on this page?