# Publish test runs from CI

Run your Playwright specs against a deployment on every push and publish the run, with a retry for flaky tests.

This example runs the storefront's Playwright specs against its staging deployment on every push to
`main`, publishes the run to the Acme workspace, and retries failed tests once so a flaky test does
not fail the build. It uses GitHub Actions; any CI that runs a shell works the same way.

## Before you begin

- The `storefront` product is deployed to `staging` with a base URL. See
  [Record a deployment from CI](https://docs.qa.clankership.net/get-started/examples/record-a-deployment.md).
- The repository has Playwright specs and a Playwright config. The project does not need
  Playwright installed: `clanker run` brings its own.
- You are an admin or owner of the workspace, to create the account API key.

## 1. Link the repository

Run this once on your computer, from the repository root, and commit the file it writes:

```bash
clanker link acme --product storefront
git add clanker.json
git commit -m "Link the repository to Clanker Ship"
```

`clanker.json` names the workspace by ID and the product the tests cover, so CI needs no flags for
them.

## 2. Create an account API key

An account API key belongs to the workspace, so the pipeline keeps working when people leave. Give
it the member role, which can publish and retry runs. Create it on **Settings › Security › Account
API keys**, or with [Create an account API key](https://docs.qa.clankership.net/api/account-api-keys/create.md):

```bash
curl https://api.clankertest.com/v1/workspaces/acme/account-api-keys \
  -H "Authorization: Bearer $CLANKER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "name": "Storefront CI", "role": "member" }'
```

Copy the `secret` from the response; it is shown once. Store it in your CI as a secret named
`CLANKER_API_KEY`. In GitHub, that is **Settings › Secrets and variables › Actions** in the
repository.

## 3. Add the workflow

```yaml
# .github/workflows/validation-tests.yml
name: Validation tests

on:
  push:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install the Clanker CLI
        run: |
          curl -fsSL https://clankertest.com/install.sh | sh
          echo "$HOME/.local/bin" >> "$GITHUB_PATH"

      - name: Install Chrome's system libraries
        run: clanker install-deps

      - name: Run and publish the tests
        env:
          CLANKER_API_KEY: ${{ secrets.CLANKER_API_KEY }}
        run: clanker run --environment staging || clanker retry
```

If your specs import packages of their own, such as `dotenv`, install the project's dependencies
before the run step.

## What happens

1. `clanker run --environment staging` reads the storefront's staging deployment, starts a run,
   runs the specs against the deployment's base URL, uploads each test's video, screenshots and
   trace, and finishes the run. It prints the run's URL.
2. It exits `0` when every test passed, so the step ends there.
3. When a test failed, it exits `1` and `clanker retry` runs the failed tests again on the same
   commit, twice each by default. A test that passes every retry attempt counts as flaky, and the
   retry exits `0` when no test is still failing.
4. When the run could not start or publish, such as when the product is not deployed to staging,
   it exits `2`. The retry then finds no run on this commit and fails too, so the build fails.

The run shows on the workspace's **Test runs** page with its commit and the account API key that
published it, and a run with a flaky test shows as `Passed*`.

## Variations

- **Test a preview build.** Pass the preview's URL; the run keeps the deployment's URL too, and the
  website marks the override:

  ```bash
  clanker run --environment staging --base-url "https://pr-482.storefront.acme.dev"
  ```

- **Run only the critical cases.** `clanker run --environment staging --coverage critical`.
- **Require more passes to call a test flaky.** `clanker retry --passes 3`.
- **Find failed runs from a script.** [List test runs](https://docs.qa.clankership.net/api/test-runs/list.md) with `result=failed`
  and `commit=<sha>`.

See [Test runs](https://docs.qa.clankership.net/get-started/concepts/test-runs.md) for how runs, retries and verdicts work.