I've worked on teams where the response to a flaky test was to rerun the CI pipeline until it passed. Sometimes we'd hit the 'Re-run' button four or five times before the test decided to cooperate. That's not debugging—that's gambling. And it erodes trust in the test suite faster than any genuine failure.
Flaky tests are not supernatural. They have a root cause, and it's almost always something we can find and fix. The problem is that most of us treat flaky tests as random noise rather than signals. In this post, I'll walk through the systematic approach I've developed over years of debugging flaky tests—classifying the failure, reproducing it reliably, isolating the root cause, and building a fix that stays fixed.
Classify the Flake: Four Common Patterns
Every flaky test I've ever debugged falls into one of four categories. Once you know which category you're dealing with, you're halfway to the fix.
- arrow_rightOrdering dependencies: The test passes when run in isolation but fails when run after another test. This is almost always due to shared state (static variables, database records, file system).
- arrow_rightTiming/race conditions: The test relies on an asynchronous operation completing within a certain time window. Common in UI tests, network calls, and multi-threaded code.
- arrow_rightResource exhaustion: The test fails only when the system is under load—CPU throttling, memory limits, or connection pool depletion.
- arrow_rightExternal dependency flakiness: The test calls a real API, database, or service that occasionally times out or returns an unexpected response.
When you first encounter a flaky test, don't try to fix it yet. Instead, collect data: check if the failure is always the same assertion, whether it correlates with other tests, and whether it happens more often at certain times of day (which might indicate resource contention).
Reproduce the Flake: Stress It Until It Breaks
The hardest part of debugging flaky tests is reproducing them. A test that fails 1% of the time might require hundreds of runs to see a failure. You can't bisect or add instrumentation if you can't trigger the bug.
The trick is to amplify the conditions that cause the flake. If you suspect a race condition, run the test in a loop with multiple threads. If you suspect ordering, run the entire test suite in random order. If you suspect timing, add randomized delays or throttle the CPU.
# Loop the test 100 times to increase reproduction rate
for i in $(seq 1 100); do
python -m pytest tests/test_checkout.py -x --random-order
if [ $? -ne 0 ]; then
echo "Failed on iteration $i"
break
fi
doneIf that doesn't work, I use a trick I call 'race amplification': I wrap the asynchronous operation in a helper that randomly injects delays. This simulates the worst-case timing without changing the production code.
import random
import asyncio
async def flaky_operation(delay=0.5):
# Simulate network call
await asyncio.sleep(delay)
# Randomly inject extra delay to surface race conditions
if random.random() < 0.1:
await asyncio.sleep(random.uniform(0.1, 0.5))
return fetch_data()A Real-World Example: The Case of the Phantom 503
Last year, I was debugging a test that would fail about once every 50 CI runs. The assertion was always the same: the response status code was 503 instead of 200. But the API endpoint was working fine—I could hit it from my machine and get 200 every time.
I started by checking the CI environment. The test was making a real HTTP call to a staging service that had rate limiting. In CI, multiple test suites ran in parallel, and sometimes they'd collectively exceed the rate limit. The 503 was coming from the API gateway, not the service itself.
The fix was to either mock the API call or to use a dedicated CI test account with a higher rate limit. We chose to mock the call and add a contract test that verified the real API behavior separately.
The Rate Limit Flake
- Day 1Test checkout fails with 503 in CI. Rerun passes. Marked as flaky.
- Day 3Same test fails again. Team reruns and moves on.
- Day 7Fails three times in one day. I start investigating.
- Day 8Check CI logs: all failures happen during peak parallel test execution.
- Day 9Found that staging API has a rate limit of 100 req/min. CI runs 150 req/min during peak.
- Day 10Fix: mock the API in unit tests, add separate contract test for rate limiting.
Lesson
External dependencies are the most common source of flakiness in integration tests. Always check rate limits, timeouts, and concurrent usage in CI.
Instrument the Test: Add Diagnostics That Don't Change Behavior
Once you can reproduce the flake, you need to understand exactly what's happening at the moment of failure. A good assertion message isn't enough—you need to see the state of the system before and during the failure.
I add structured logging to the test itself. Not print statements—actual structured logs with timestamps, thread IDs, and relevant state. The key is to make sure the instrumentation doesn't change the timing or behavior of the test.
import logging
import threading
logger = logging.getLogger(__name__)
def test_checkout():
logger.info("Starting test_checkout", extra={"thread": threading.get_ident()})
# Setup
user = create_user()
logger.debug(f"User created: {user.id}")
# Action that sometimes fails
result = perform_checkout(user)
logger.info(f"Checkout result: {result.status_code}", extra={"body": result.text[:200]})
# Assertion
assert result.status_code == 200, f"Expected 200, got {result.status_code}: {result.text}"Be careful with logging in performance-sensitive tests. If your logging adds significant latency, it might mask the very race condition you're trying to catch. Use async logging or buffered loggers to minimize overhead.
Isolate the Root Cause: Use Scientific Method
With reproducible failures and good instrumentation, you can start isolating the root cause. I use a hypothesis-driven approach: come up with a theory, change one variable, and see if the failure rate changes.
For example, if I suspect a shared static variable, I'll add a reset in the test setup and see if the flake disappears. If I suspect a timing issue, I'll add a small sleep before the assertion and see if the failure rate drops.
- 1Form a hypothesis about the root cause (e.g., 'The test fails because of leftover data from a previous test').
- 2Change one variable (e.g., add a cleanup step in setup).
- 3Run the test 100 times and record the failure count.
- 4If the failure count drops to zero, you've found the cause. If not, revert and try the next hypothesis.
- 5Document the root cause and the evidence you collected.
This sounds obvious, but I've seen people throw random fixes at flaky tests without any systematic testing. They'll add a retry loop, increase a timeout, or reset some state and hope it works. That's not debugging—it's superstition.
Fixing the Root Cause: Three Strategies
Once you know the root cause, the fix should be surgical. Here are the three strategies I use most often:
- arrow_rightIsolation: Make the test truly independent. Use database transactions that roll back after each test, reset static state in setup, and avoid sharing fixtures between tests.
- arrow_rightSynchronization: For race conditions, use proper synchronization primitives (locks, barriers, or promises) instead of sleeps. If the test must wait for an async operation, use an explicit wait with a timeout and a condition.
- arrow_rightMocking: For external dependencies, mock the boundary and write separate integration tests that can tolerate flakiness. This reduces the surface area for flakiness without losing coverage.
# Before: flaky test relying on timing
@pytest.mark.asyncio
async def test_notification_sent():
await send_notification()
await asyncio.sleep(1) # fragile sleep
assert get_notification_count() == 1
# After: explicit wait with condition
@pytest.mark.asyncio
async def test_notification_sent():
await send_notification()
await wait_for_condition(
lambda: get_notification_count() == 1,
timeout=5.0,
interval=0.1
)
assert get_notification_count() == 1Build a Flaky Test Triage Pipeline
No matter how careful you are, flaky tests will still appear. The key is to catch them early and route them to the right people. I recommend building a simple pipeline that runs tests multiple times and flags any test that fails inconsistently.
At Buglyst, we run each test in the CI pipeline three times. If a test passes at least once but not all three, it's quarantined and a ticket is automatically created for the owning team. The test is still run in a separate job to track its behavior, but it doesn't block deployments.
Reduction in deployment-blocking flaky tests after implementing triage pipeline
The triage pipeline also collects metadata: which commit introduced the flake, which environment variables were set, and which other tests were running concurrently. This data makes root cause analysis much faster.
The Final Step: Make the Fix Stick
I've seen teams fix a flaky test, only to have the same root cause pop up in a different test two weeks later. That's because they fixed the symptom, not the systemic issue.
After you fix a flaky test, write a regression test that explicitly checks for the root cause. For example, if the flake was due to shared static state, write a test that runs two tests that share that state and verifies they don't interfere. This prevents the same class of bug from recurring.
And finally, add a monitoring alert. If the same test (or a similar one) starts flaking again, you want to know immediately. Flaky tests are like cockroaches—if you see one, there are probably more hiding.
A flaky test that's been 'fixed' three times is a sign that your team doesn't understand the root cause. Stop patching and start investigating—the fourth fix might actually work.
Summary
- arrow_rightClassify the flake: ordering, timing, resource, or external dependency.
- arrow_rightReproduce by amplifying conditions: loop, randomize, add delays.
- arrow_rightInstrument with structured logging that doesn't alter behavior.
- arrow_rightIsolate root cause using the scientific method—change one variable at a time.
- arrow_rightFix surgically: isolate, synchronize, or mock.
- arrow_rightBuild a triage pipeline to catch flaky tests early.
- arrow_rightWrite regression tests for the root cause and monitor for recurrence.
Flaky tests are not a fact of life. They're a bug in your test suite, and they deserve the same rigorous debugging as any production bug. The next time you see a test fail and then pass on a rerun, don't just click the button—start investigating. The root cause is waiting to be found.
Frequently asked questions
What is the most common root cause of flaky tests?
In my experience, shared mutable state between tests is the most common culprit. This includes static variables, test fixtures that aren't properly isolated, and global configuration that gets modified but not reset. The second most common is race conditions from improper synchronization in async code.
How do I reproduce a flaky test that only fails in CI?
You need to replicate the CI environment as closely as possible. Start by running the test in a Docker container with the same CPU/memory limits. Then apply stress techniques: run the test in parallel with other tests, add random delays, or use tools like chaos monkey to simulate resource contention. Also check if CI has different environment variables or file system behavior.
Should I just disable flaky tests or quarantine them?
Quarantine is better than disabling. Move the test to a separate CI job that runs after the main pipeline, and notify the owning team when it fails. This prevents blocking deployments while still tracking the failure. But you should still fix the root cause within a sprint—quarantine is not a permanent solution.
What tools can help detect flaky tests automatically?
For Java/Kotlin, there's NonDex to detect non-deterministic behavior. For Python, pytest-flaky can rerun tests and report flakiness. CI platforms like Buildkite and CircleCI have built-in flaky test detection. I also recommend using test ordering randomization (e.g., pytest-randomly) to surface ordering dependencies.