How Do You Stop Flaky Tests From Breaking Your CI Pipeline?
The fastest way to stop flaky tests from breaking your CI pipeline is to **detect them automatically, quarantine them out of the blocking path, and fix the root cause on a deadline** — rather than letting engineers re-run failed jobs until they go green. A re-run culture trains your whole team to ignore red builds, which defeats the point of having tests at all.
## What Makes a Test Flaky
A flaky test is one that passes and fails on the same code without any change. The common causes are surprisingly few:
- **Timing and async races** — asserting before an element renders or a promise resolves.
- **Shared state** — tests that depend on order, or that read and write the same database row or global variable.
- **External dependencies** — real network calls, third-party APIs, or system clocks.
- **Non-deterministic data** — random seeds, unsorted collections, or time-of-day logic.
- **Resource contention** — parallel tests competing for ports, files, or memory under load.
If you cannot explain why a test failed in one sentence, treat it as flaky until proven otherwise.
## Detect Flakiness Automatically
You cannot manage what you cannot see. Add detection before you add policy:
1. **Record every test result over time**, not just the final job status. Store test name, outcome, duration, and commit.
2. **Flag tests that fail then pass on a re-run of the identical commit.** That signature is the clearest evidence of flakiness.
3. **Track a flake rate per test** — the percentage of runs that disagree with the majority result.
4. **Surface the worst offenders** in a weekly report so they cannot hide in a 4,000-test suite.
Many CI platforms now offer built-in flaky-test analytics; if yours does not, a small script parsing JUnit XML output into a database is enough to start.
## Quarantine Without Hiding
Once a test is confirmed flaky, move it out of the blocking path so it stops failing honest builds — but never simply delete it or mark it skipped forever.
- Tag it (for example `@quarantine`) and run it in a **separate, non-blocking job**.
- Keep collecting its results so you know when it has stabilised.
- Set an **explicit expiry**: a quarantined test that is not fixed within, say, two weeks gets escalated or deleted, with the gap in coverage logged as a known risk.
Quarantine is a holding pen, not a graveyard. The danger is a suite where hundreds of tests are permanently quarantined and nobody remembers what they covered.
## Fix the Root Cause
Quarantine buys time; it does not fix anything. When you return to a flaky test:
- Replace fixed `sleep` calls with **explicit waits on a condition**.
- Isolate state — give each test its own database transaction, fixture, or namespace, and roll back afterwards.
- Stub external services so the test does not depend on someone else's uptime.
- Freeze the clock and seed randomness so runs are reproducible.
- If a test is flaky because the *feature* is genuinely racy, that is a product bug, not a test bug — fix the code.
## Make Green Mean Something
The cultural goal is simple: a green build should mean the change is safe, and a red build should mean stop. Re-running until green destroys that signal. Protect it by:
- **Banning blind re-runs** as a debugging strategy; require a quarantine tag or a fix instead.
- **Alerting on flake-rate regressions** the same way you alert on coverage drops.
- **Owning tests like code** — every test has a team responsible for its reliability.
This discipline is what separates a test suite that catches real regressions from one that everyone has learned to ignore. At neart.ai we build enterprise-grade quality tooling precisely because reliable signal at scale is hard to achieve by hand.
## Practical Takeaway
Don't fight flakiness by re-running jobs. Instrument your suite to detect flaky tests automatically, quarantine confirmed offenders into a non-blocking job with a hard expiry date, and fix the root cause — timing, shared state, or external calls — so that green always means safe to ship.