Test runs

A test run is one clanker run of your Playwright specs, published to your workspace. Every member can open it on the website: each test, every attempt, and its error, screenshots, video and trace.

• A run tests one product in one environment. The CLI reads that deployment's baseUrl from your workspace, so you name only the environment.
• Each test is passed, flaky, failed or skipped. Attempt 1 decides passed, failed or skipped.
• A run is failed when a test failed, or when Playwright reported an error outside the tests, such as a web server that did not start. The run page shows that error. A run with no tests never passes.
• clanker retry runs a run's failed tests again on the same commit. A test that passes every retry attempt (two by default) is flaky and counts as passed. The run shows an asterisk: Passed*, 100%*.
• Pass rate is (passed + flaky) ÷ (tests − skipped). The first-attempt pass rate never changes.

Publish a run

Link the repository once, with the product its tests cover:

clanker link acme --product storefront

Then run against an environment:

clanker run --environment qa
clanker run --environment preview --base-url https://pr-482.storefront.acme.dev
clanker run --environment qa --coverage critical

--base-url replaces the deployment's URL for one run, for a local server on another port or a preview build. --product tests another product of the workspace. --no-publish runs without publishing; without --environment it uses your Playwright config's own baseURL and needs no sign-in.

The CLI prints the run's URL. It exits 0 when every test passed, 1 when the run failed, and 2 when the run could not start or publish, for example when the product is not deployed to that environment. Each evidence file is tried up to five times when the connection drops or the server has an error; a file that still fails stops the publish, and the CLI names it.

Retry failures

clanker retry 1289
clanker retry --passes 3
clanker run --environment qa || clanker retry

A retry runs only failed tests, against the run's URL, on the commit the run tested; on another commit it is refused, because a pass on changed code is a fix, not flakiness. Without a number it retries your newest run on the current commit. It exits 0 when no test is still failing.

Rules

• Members, admins and owners publish and retry; viewers read. Account API keys use their role, so CI can publish with CLANKER_API_KEY.
• Only finished runs with failed tests and no error outside them can be retried, one retry at a time.
• A run still running after 6 hours is marked interrupted.
• Files are at most 100 MB each.

HTTP

Paths are relative to /v1/workspaces/{slug}; {slug} may also be the workspace id, and {run} is the run number. See the OpenAPI reference.