# Test runs

A **test run** is one `clanker run` of your Playwright specs, published to your
workspace. Every member can open it on the website: each test, every attempt,
and its error, screenshots, video and trace.

- A run tests one **product** in one **environment**. The CLI reads that
  deployment's `baseUrl` from your workspace, so you name only the environment.
- Each test is **passed**, **flaky**, **failed** or **skipped**. Attempt 1
  decides passed, failed or skipped.
- A run is **failed** when a test failed, or when Playwright reported an error
  outside the tests, such as a web server that did not start. The run page shows
  that error. A run with no tests never passes.
- `clanker retry` runs a run's failed tests again on the same commit. A test
  that passes every retry attempt (two by default) is **flaky** and counts as
  passed. The run shows an asterisk: `Passed*`, `100%*`.
- **Pass rate** is (passed + flaky) ÷ (tests − skipped). The first-attempt pass
  rate never changes.

## Publish a run

Link the repository once, with the product its tests cover:

```bash
clanker link acme --product storefront
```

Then run against an environment:

```bash
clanker run --environment qa
clanker run --environment preview --base-url https://pr-482.storefront.acme.dev
clanker run --environment qa --coverage critical
```

`--base-url` replaces the deployment's URL for one run, for a local server on
another port or a preview build. `--product` tests another product of the
workspace. `--no-publish` runs without publishing; without `--environment` it
uses your Playwright config's own `baseURL` and needs no sign-in.

The CLI prints the run's URL. It exits 0 when every test passed, 1 when the run
failed, and 2 when the run could not start or publish, for example when the
product is not deployed to that environment. Each evidence file is tried up to
five times when the connection drops or the server has an error; a file that
still fails stops the publish, and the CLI names it.

## Retry failures

```bash
clanker retry 1289
clanker retry --passes 3
clanker run --environment qa || clanker retry
```

A retry runs only failed tests, against the run's URL, on the commit the run
tested; on another commit it is refused, because a pass on changed code is a
fix, not flakiness. Without a number it retries your newest run on the current
commit. It exits 0 when no test is still failing.

## Rules

- Members, admins and owners publish and retry; viewers read. Account API keys
  use their role, so CI can publish with `CLANKER_API_KEY`.
- Only finished runs with failed tests and no error outside them can be
  retried, one retry at a time.
- A run still running after 6 hours is marked interrupted.
- Files are at most 100 MB each.

## HTTP

Paths are relative to `/v1/workspaces/{slug}`; `{slug}` may also be the
workspace id, and `{run}` is the run number. See the
[OpenAPI reference](/openapi.json).

| Operation       | HTTP                                                      |
| --------------- | --------------------------------------------------------- |
| List runs       | `GET /test-runs?product=&environment=&result=&q=&cursor=` |
| Get a run       | `GET /test-runs/{run}`                                    |
| Start a run     | `POST /test-runs`                                         |
| Upload a file   | `PUT /test-runs/{run}/files/{path}`                       |
| Finish a run    | `POST /test-runs/{run}/finish`                            |
| Start a retry   | `POST /test-runs/{run}/retries`                           |
| Finish a retry  | `POST /test-runs/{run}/retries/{retry}/finish`            |
| Download a file | `GET /test-runs/{run}/files/{path}`                       |