CI/CD Integration

CI/CD integration

Run an eval on every deploy to catch regressions before they reach production. Call the API from GitHub Actions, GitLab CI or any other CI system.

How it works

  1. Create a collection with your stores, models and sequences (once)
  2. Start a collection run with POST /api/v1/collections/{id}/run
  3. Poll until it finishes with GET /api/v1/collection-runs/{run_id}
  4. Check the results: assert on checkout rate, error count or duration

GitHub Actions example

name: UCP Store Evals
on:
  deployment_status:
    types: [completed]

jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - name: Run collection
        run: |
          RUN_ID=$(curl -s -X POST \
            -H "Authorization: Bearer ${{ secrets.UCP_TOKEN }}" \
            https://ucpplayground.com/api/v1/collections/${{ vars.COLLECTION_ID }}/run \
            | jq -r '.run_id')

          # Poll until complete (max 5 minutes)
          for i in $(seq 1 30); do
            STATUS=$(curl -s \
              -H "Authorization: Bearer ${{ secrets.UCP_TOKEN }}" \
              https://ucpplayground.com/api/v1/collection-runs/$RUN_ID \
              | jq -r '.status')
            [ "$STATUS" = "complete" ] && break
            sleep 10
          done

          # Check checkout rate
          RATE=$(curl -s \
            -H "Authorization: Bearer ${{ secrets.UCP_TOKEN }}" \
            https://ucpplayground.com/api/v1/collection-runs/$RUN_ID \
            | jq '.summary.checkout_rate')

          echo "Checkout rate: $RATE%"
          if (( $(echo "$RATE < 80" | bc -l) )); then
            echo "::error::Checkout rate $RATE% below 80% threshold"
            exit 1
          fi

API endpoints

EndpointDescription
POST /api/v1/collectionsCreate a collection
GET /api/v1/collectionsList collections
POST /api/v1/collections/{id}/runStart a collection run
GET /api/v1/collection-runs/{id}Get a collection run's status and summary
GET /api/v1/collection-runs/{id}/pdfDownload the PDF report
Tip

Keep your collection ID as a CI variable and your API token as a secret. The collection stays between deploys: create it once, then start a collection run on each deploy.

Scheduled runs

To keep checking for regressions, set a cron schedule on the collection. Collection runs then start on their own while is_active is true:

curl -X PATCH -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  https://ucpplayground.com/api/v1/collections/{id} \
  -d '{ "schedule": "0 9 * * 1" }'  # Every Monday at 9am

The API rejects invalid cron expressions. List past collection runs with GET /api/v1/collections/{id}/runs to follow the trend. If you would rather start them from your own CI, a scheduled GitHub Actions workflow calling POST /api/v1/collections/{id}/run works just as well.

Completion webhooks

Set a webhook_url (HTTPS only) on the collection to get a POST when each collection run finishes, with no polling:

curl -X PATCH -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  https://ucpplayground.com/api/v1/collections/{id} \
  -d '{ "webhook_url": "https://ci.example.com/hooks/ucp" }'

The response includes a webhook_secret. It is shown once, and changes whenever the URL changes. Each delivery is signed with X-Playground-Signature: sha256=<HMAC-SHA256 of the raw body>; check it before you trust the payload. The body carries event: collection_run.completed, the collection run's id and status, and its full summary (checkout rates, breakdowns by store and by model, errors). A failed delivery is retried twice, with a wait between tries.

Assertions

Set thresholds in the collection's config and we check them when each collection run finishes. The summary carries passed_assertions (the webhook payload has it too), so CI can gate on one true-or-false value:

"config": {
  "assertions": { "min_checkout_rate": 80, "max_avg_tokens": 50000, "max_errors": 0 }
}

Supported: min_checkout_rate, min_search_rate, max_avg_tokens, max_avg_duration_ms, max_errors. The API rejects any other key. Each assertion's threshold and actual value are in summary.assertions.

What to assert on

  • checkout_rate >= 80: a store ready for production should reach at least 80%
  • errors.total == 0: no tool errors means the MCP side is clean
  • avg_duration_ms < 30000: runs over 30 seconds can point to a slow MCP endpoint
  • avg_tokens < 50000: high token use points to long product descriptions or an agent going round in circles

Something wrong on this page?