Test runs

A test run is one clanker run of a project's Playwright specs, published to your workspace. Every member can open it on the website: each test, every attempt, and its error, screenshots, video and trace. clanker retry runs a run's failed tests again on the same commit and adds the attempts to the same run, so a test that failed only once counts as flaky instead of failed.

Objects

A run keeps the URL it tested and the deployment's URL and version when it started, so changing the deployment later does not change the run. When the two URLs differ, the run used --base-url and the website marks the URL as an override.

A test is identified by its case ID: the spec's @<id> tag, or an ID made from its file and title. A spec that runs in several Playwright projects is one test per project.

Verdicts

• Attempt 1 decides a test's status: passed, skipped, or failed for anything else.
• A failed test becomes flaky when one retry ran it passes times and every attempt passed. A failed attempt leaves it failed, and a later retry can still flip it.
• Pass rate is (passed + flaky) ÷ (tests − skipped). The first-attempt pass rate counts attempt 1 only and never changes after the run finishes.
• A run's error is what failed outside its tests, as Playwright reported it: a web server that did not start, a global setup error, no tests found, or a config that did not load. A finished run with no tests gets the error No tests ran., because a run that tested nothing proves nothing.
• A run passed when it has no error and no failed test. It shows an asterisk, such as Passed* and 100%*, when a test is flaky. A run with an error is failed even when every test passed.

clanker run turns Playwright's own retries off, so attempt 1 is always the first real result and every retry shows on the run page.

Publish a run

Link the repository once, with the product its tests cover, and commit clanker.json:

clanker link acme --product storefront

Then run the specs against an environment. The CLI reads that deployment's baseUrl from the workspace, so you name only the environment:

clanker run --environment staging
clanker run --environment staging --coverage critical
clanker run --environment staging --base-url http://localhost:4173

The CLI uses its own Node, Playwright and Chrome, so the project does not need Playwright installed. It prints the run's URL and exits:

Evidence uploads six files at a time. A file is tried up to five times when the connection drops or the server answers 5xx, 408 or 429. A file that still fails stops the publish: the CLI names the file and exits 2, and the run stays running until it is marked interrupted.

Test data

Every case should run in every environment, production included, so a run makes the data its tests need and deletes all of it when it ends. The product exposes a Fixture API: routes that create test data, such as a verified user with a role, and delete it. A deployment's fixtureUrl says where it is.

When the deployment has a fixture URL, starting a run creates a scope, such as scp_7h2k9q2m4x1a, and returns a fixture token for it. Specs read both from Playwright's clanker option, with the environment and every product's URL there, and send the token to the Fixture API to make data in the scope. When Playwright exits, Ctrl-C included, the CLI deletes the scope and reports what the teardown said with the finish:

A retry gets a new scope, because the run's is gone by then. A scope expires six hours after it starts, and a later teardown on the deployment deletes whatever an expired scope left. A fixture token is an EdDSA JWT for one scope and one Fixture API, which checks it against GET /.well-known/jwks.json on the Clanker Ship API.

Cases tagged @local are left out of runs away from your machine, and @non-production cases out of production runs. Check that a Fixture API creates and deletes every fixture it lists with:

clanker fixtures check --environment staging

Semaphores

Some resources stay shared across scopes: one sandbox account, a vendor API with a rate limit. A semaphore limits how many tests use one at a time, across every worker and every run in the workspace. Each published run and retry gets a run token in Playwright's clanker option. A test takes a lease with POST {semaphores.url}/{name}/leases, sending the semaphore's capacity, renews it while it works and releases it when done. The first lease creates the semaphore, a lease lasts its TTL, 5 to 600 seconds and 60 by default, unless renewed, and finishing a run releases what is left. clanker semaphore list and Settings › Semaphores show who holds what.

Retry failures

clanker retry 1289
clanker retry --passes 3
clanker run --environment staging || clanker retry

A retry runs only the failed tests, against the run's URL, on the commit the run tested. On another commit it is refused, because a pass on changed code is a fix, not flakiness. Without a number it retries your newest run on the current commit, so CI can chain run || retry.

Only a finished run with failed tests and no error can be retried, one retry at a time. A run still running after 6 hours is marked interrupted, and an interrupted run cannot be retried.

Who can do what

Account API keys act with their role, so CI publishes with a member key in CLANKER_API_KEY; see Publish test runs from CI. Only the publisher of a run or a retry uploads its files and finishes it.

API

Paths are relative to https://api.clankertest.com/v1/workspaces/{slug}, and {run} is the run's number. clanker run makes these calls for you.

Lists are newest first, 50 runs per page; see Pagination. result=flaky lists runs with at least one flaky test. File paths are results/{case}/case.md, results/{case}/spec.ts and results/{case}/attempts/{n}/video.webm, trace.zip or screenshot-{k}.png, at most 100 MB each.

curl "https://api.clankertest.com/v1/workspaces/acme/test-runs?product=storefront&result=failed" \
  -H "Authorization: Bearer $CLANKER_API_KEY"

Errors