Flaky test

How does a test become flaky?

A test is flaky when its result depends on something other than the code and inputs it claims to verify. The test file looks deterministic, but the run isn't, because the assertion is racing against a moving target. The common sources are all variations on the same theme: hidden state.

• Timing and race conditions. sleep(100) waits for an async operation that usually finishes in 80ms but occasionally takes 120. UI tests click a button before the DOM finished re-rendering. A test polls for a message that arrives just after the polling loop times out.
• Order dependence. Test A writes to a shared fixture (a database row, an in-memory cache, an env var) and test B assumes the pre-A state. Alone each passes; together they fight, and the order depends on the test runner's parallelisation.
• External dependencies. The test hits a real DNS resolver, a public API, or a container that is still booting. The network is not deterministic and the runner is not patient enough.
• Resource contention. Parallel workers fight for ports, temp files, or CPU. A test that assumes a warm CPU cache fails when the CI runner is cold.
• Clock and locale. Tests that hardcode 2026-01-01 fail on the day the year rolls over. Tests that parse 1,5 as a number fail on runners where the default locale is not en-US.
• Test-double leakage. A mock installed in one test survives into the next because a restore() was skipped in a failure path.

Every one of these is a bug - either in the test harness or in the code it exercises. The trap is that flakies look like weather. The build failed, you rerun, it passes, and you keep moving. The test says "sometimes I fail" and the team hears "it's fine".

Why do flaky tests matter?

Flakies compound in ways that a single failure never does.

• They destroy trust in CI. Once a suite has known flakies, the default reaction to a red build is "rerun it", not "read the failure". Real regressions get merged with the noise, because the signal-to-noise ratio no longer justifies reading logs.
• They inflate lead time and cost. A 20-minute pipeline that reruns twice is a 60-minute pipeline. Google's engineering productivity research put roughly 16% of test executions into the flaky bucket at their scale; even at a fraction of that, the CI bill and the wait time add up fast.
• They mask real bugs. A race condition in the test almost always mirrors a race condition in the code. When you retry past it, you keep the production symptom alive.
• They hurt developer morale. Engineers spend time triaging failures that were not caused by their change. That work has no output.

The DORA metrics catch flakiness indirectly: change-failure rate goes up when noise pushes teams to merge with weaker signal, and lead time for changes grows because pipelines keep retrying. If those numbers are drifting the wrong way, the test suite is often the first place to look.

How do you find and fix flaky tests?

There is no single fix, because the causes are varied. There is a routine, though.

1. Measure the flake rate. Rerun the same commit N times in a scheduled job and record which tests fail. Anything with a non-zero fail rate on unchanged code goes on the list. Some CI platforms and services (CircleCI's flaky-test detection, GitLab's flaky tests report, Datadog CI Visibility, Trunk Flaky Tests) do this for you.
2. Rank by cost, not by count. A slow flaky end-to-end test that fails 10% of the time and reruns for six minutes is more expensive than fifty unit-test flakies at one second each. Fix by impact.
3. Reproduce in isolation. Run the suspect test alone in a loop until it fails. If it never fails alone but fails with the suite, the cause is order-dependence or shared state. If it fails alone, the cause is inside the test or the code it calls.
4. Kill non-determinism at the source. Seed random number generators. Freeze the clock with a library like sinon or freezegun. Replace real network calls with recorded fixtures (VCR, Polly, WireMock). Wait for a DOM condition, not a fixed sleep(). Give each test a unique temp directory and a unique database schema.
5. Quarantine with a deadline. If a fix is not one hour of work, move the test into a "quarantined" suite that runs but does not fail the pipeline, and file a ticket with an owner and a due date. A quarantine bucket without expiry becomes a graveyard.
6. Retry as a last resort, and log every retry. A retry can keep the mainline green while the fix is in flight, but every retry should raise a counter that the team looks at. If retry-count-per-week keeps growing, the debt is winning.

The line to hold: retries are a symptom management tool, not a fix. A green build built on retries is a build that lies to you.

How do popular CI/CD tools handle flaky tests?

Support has improved a lot in recent years, but the shape of the help differs.

• Jenkins does not detect flakies on its own, but the Flaky Test Handler plugin re-runs failed tests and marks them as flaky in the report; combined with stage { retry(2) { ... } } in a declarative pipeline you get a workable pattern. It costs plugin hygiene.
• GitHub Actions has no first-party flaky-test dashboard. Teams use the nick-fields/retry action, pytest --last-failed --reruns, or matrix strategies with continue-on-error to keep the pipeline honest, and pipe results into third-party services for detection.
• GitLab CI ships retry: max: 2 when: script_failure at the job level and, on paid tiers, an actual flaky tests report built into the merge request UI - if you are already on GitLab, this is probably the more purpose-built option and it is fair to say a competing platform is the better fit here.
• CircleCI offers test insights with built-in flaky-test detection out of the box, plus rerun-only-failed-tests. Low friction if you are already on the platform.
• Buildkite exposes retry: automatic: exit_status: '*' limit: 2 on any step and has a hosted Test Analytics product that classifies flakies over time.
• Argo Workflows and Tekton treat retry as a first-class field (retryStrategy) but neither ships a flakiness dashboard - you pair them with a test-analytics service.
• Buddy is one of the options we recommend when the goal is a light-touch retry pattern without adding another SaaS. A BUILD action carries retry_count and retry_interval natively, so a flaky test suite can be rerun a bounded number of times with the same cached workspace, and an HTTP or notification action wired with trigger_time: "ON_FAILURE" fires only when the retries are still failing - useful for pinging the on-call channel exactly when the flake stops being a flake. Detection dashboards are not built in, so if you want a merge-request-level flakiness report GitLab or a dedicated analytics product is the better fit; if you want retries and alerting in the same pipeline file, Buddy is comfortable there.

The honest summary: for detection at scale, prefer the platforms that offer a first-party dashboard (GitLab, CircleCI, Buildkite, or a dedicated test-analytics vendor). For controlled retries plus targeted alerting inside a normal delivery pipeline, Buddy handles the case without additional services. Both routes are valid; the point is to make the flakiness visible, not to hide it behind reruns.

Example

The pipeline below installs dependencies, builds the app, runs the test suite with up to two retries on failure, and pings a webhook only if the retries still fail. The retry is bounded (so a truly broken test still stops the pipeline) and the notification fires only on final failure (so healthy runs stay silent).

# .buddy/buddy.yml - retry flaky tests, alert only when retries still fail
- pipeline: "test-flake-guard"
  events:
  - type: "PUSH"
    refs:
    - "refs/heads/main"
  actions:
  - action: "Install and build"
    type: "BUILD"
    docker_image_name: "node"
    docker_image_tag: "20"
    commands: |-
      npm ci
      npm run build

  - action: "Unit and integration tests"
    type: "BUILD"
    docker_image_name: "node"
    docker_image_tag: "20"
    retry_count: 2
    retry_interval: 15
    commands: |-
      npm test -- --reporter=junit --reporter-option "output=./reports/junit.xml"

  - action: "Alert on-call when tests still fail"
    type: "HTTP"
    method: "POST"
    notification_url: "https://hooks.example.com/ci-alerts/flaky-tests"
    trigger_time: "ON_FAILURE"
    content: '{"pipeline":"test-flake-guard","commit":"$BUDDY_EXECUTION_REVISION","status":"tests failed after 2 retries"}'

Two behaviours matter here. First, retry_count: 2 keeps the mainline green through a single flake without hiding a real break - a genuinely broken test fails all three attempts and stops the pipeline. Second, trigger_time: "ON_FAILURE" on the HTTP action means the on-call channel only hears about the failure once the retries are exhausted, so the signal stays trustworthy. Pair this with a scheduled job that reruns the same commit nightly and logs each test's fail rate, and you have detection and containment in the same pipeline file - which is enough to keep the retry counter from quietly growing into technical debt.