Publish test runs from CI

This example runs the storefront's Playwright specs against its staging deployment on every push to main, publishes the run to the Acme workspace, and retries failed tests once so a flaky test does not fail the build. It uses GitHub Actions; any CI that runs a shell works the same way.

Before you begin

• The storefront product is deployed to staging with a base URL. See Record a deployment from CI.
• The repository has Playwright specs and a Playwright config. The project does not need Playwright installed: clanker run brings its own.
• You are an admin or owner of the workspace, to create the account API key.

1. Link the repository

Run this once on your computer, from the repository root, and commit the file it writes:

clanker link acme --product storefront
git add clanker.json
git commit -m "Link the repository to Clanker Ship"

clanker.json names the workspace by ID and the product the tests cover, so CI needs no flags for them.

2. Create an account API key

An account API key belongs to the workspace, so the pipeline keeps working when people leave. Give it the member role, which can publish and retry runs. Create it on Settings › Security › Account API keys, or with Create an account API key:

curl https://api.clankertest.com/v1/workspaces/acme/account-api-keys \
  -H "Authorization: Bearer $CLANKER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "name": "Storefront CI", "role": "member" }'

Copy the secret from the response; it is shown once. Store it in your CI as a secret named CLANKER_API_KEY. In GitHub, that is Settings › Secrets and variables › Actions in the repository.

3. Add the workflow

# .github/workflows/validation-tests.yml
name: Validation tests

on:
  push:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install the Clanker CLI
        run: |
          curl -fsSL https://clankertest.com/install.sh | sh
          echo "$HOME/.local/bin" >> "$GITHUB_PATH"

      - name: Install Chrome's system libraries
        run: clanker install-deps

      - name: Run and publish the tests
        env:
          CLANKER_API_KEY: ${{ secrets.CLANKER_API_KEY }}
        run: clanker run --environment staging || clanker retry

If your specs import packages of their own, such as dotenv, install the project's dependencies before the run step.

What happens

1. clanker run --environment staging reads the storefront's staging deployment, starts a run, runs the specs against the deployment's base URL, uploads each test's video, screenshots and trace, and finishes the run. It prints the run's URL.
2. It exits 0 when every test passed, so the step ends there.
3. When a test failed, it exits 1 and clanker retry runs the failed tests again on the same commit, twice each by default. A test that passes every retry attempt counts as flaky, and the retry exits 0 when no test is still failing.
4. When the run could not start or publish, such as when the product is not deployed to staging, it exits 2. The retry then finds no run on this commit and fails too, so the build fails.

The run shows on the workspace's Test runs page with its commit and the account API key that published it, and a run with a flaky test shows as Passed*.

Variations

• Test a preview build. Pass the preview's URL; the run keeps the deployment's URL too, and the website marks the override:

  bash

  clanker run --environment staging --base-url "https://pr-482.storefront.acme.dev"

• Run only the critical cases. clanker run --environment staging --coverage critical.

• Require more passes to call a test flaky. clanker retry --passes 3.

• Find failed runs from a script. List test runs with result=failed and commit=<sha>.

See Test runs for how runs, retries and verdicts work.