How Do You Triage a Flaky Test? A Step-by-Step Workflow

To triage a flaky test, work through five steps in order: confirm it is genuinely flaky by re-running it in isolation, gather evidence from logs and artefacts, classify the root cause into a known category, quarantine it so it stops blocking the pipeline, then fix and verify before returning it to the main suite. Skipping straight to a fix wastes time, because most flaky tests fail for one of a small number of well-understood reasons.

## Step 1: Confirm the flakiness

A test is flaky if it produces different results on the same code without any change. Before anything else, prove that.

- Re-run the failing test 10 to 20 times in isolation against the exact same commit.

- Record the pass/fail ratio. A test that fails 2 in 20 is flaky; one that fails 20 in 20 is simply broken and should be treated as a real failure.

- Capture the environment: browser version, parallelism, machine load, and time of day can all matter.

## Step 2: Gather evidence

Flaky tests are hard to debug precisely because they are intermittent, so you must capture everything when they do fail.

- Screenshots or video at the moment of failure.

- Full DOM snapshot and console logs.

- Network traces, especially for timing-sensitive calls.

- The test's own step-by-step trace, if your framework supports it.

Without artefacts you will be guessing, and guessing on intermittent failures is the slowest possible path.

## Step 3: Classify the root cause

Nearly every flaky test falls into one of these buckets:

- **Timing and async.** The test asserts before the app has finished updating. The most common cause by far.

- **Test order and shared state.** A test passes alone but fails when another test leaves data behind.

- **Non-deterministic data.** Random values, current dates, or unsorted results that vary between runs.

- **Environment and infrastructure.** Slow CI agents, network blips, container start-up races.

- **Animation and rendering.** Elements that are technically present but not yet interactable.

- **External dependencies.** Third-party services or APIs that are themselves unreliable.

Classifying first tells you which fix pattern to reach for, rather than fixing blindly.

## Step 4: Quarantine

A flaky test left in the blocking suite erodes trust in the entire pipeline, because engineers start ignoring red builds. Quarantine breaks that cycle.

- Move the test to a non-blocking lane that still runs and reports, but does not fail the build.

- Tag it with the suspected category and a ticket reference.

- Set an expiry. Quarantine is a holding pen, not a graveyard. A test that sits quarantined for months should be deleted or rewritten.

## Step 5: Fix by category

Apply the fix that matches the classification:

- **Timing:** replace fixed sleeps with explicit waits for a specific condition (element visible, network idle, text present).

- **Shared state:** ensure each test sets up and tears down its own data; never rely on execution order.

- **Non-deterministic data:** freeze the clock, seed random generators, and sort results before asserting.

- **Environment:** add retries only at the infrastructure boundary, not around assertions, and right-size CI resources.

- **Animation:** wait for the animation-complete state rather than a timeout.

- **External dependencies:** mock or stub them, or move the real-integration check to a separate, clearly-labelled suite.

## Step 6: Verify and return

Never declare victory after one green run. Re-run the fixed test 20+ times in the same conditions that originally exposed the flake. Only when it passes consistently should you remove it from quarantine and close the ticket.

## Make triage a habit, not a fire drill

The teams that stay on top of flakiness treat it as routine maintenance:

- Track flake rate as a first-class metric, not an anecdote.

- Assign ownership so flaky tests do not become nobody's problem.

- Review quarantine weekly to stop it filling up.

Tooling helps here. neart.ai builds enterprise-grade quality products that automate the evidence-gathering and classification steps, which are the slowest parts of triage when done by hand.

## Takeaway

Triage flaky tests with a fixed sequence: confirm, gather evidence, classify, quarantine, fix by category, then verify with repeated runs. The discipline matters more than any single tool, because most flakiness comes from a handful of predictable causes that each have a known remedy.