Flaky Tests vs Flaky Pipelines: How to Tell the Difference

A CI build fails. Someone re-runs it. It passes. Everyone moves on.

This scenario is so common that most teams have normalized it. But inside that "just re-run it" habit lies an important diagnostic question that almost nobody asks: was that a flaky test or a flaky pipeline?

The answer determines what you should fix — and fixing the wrong one wastes weeks of engineering time.

Defining the Distinction

A flaky test is a test that passes and fails non-deterministically, even in the same environment:

• Race conditions in async code
• Tests that depend on execution order
• Shared mutable state between tests
• Time-dependent assertions ("created 2 seconds ago")
• Uncontrolled randomness (random seeds, UUIDs in assertions)

A flaky pipeline is a pipeline that fails due to infrastructure or environment issues, even with perfectly stable tests:

• Network blips causing package download failures
• Runner resource contention (another job is using too much CPU/RAM)
• Cache poisoning (stale or corrupted cache restored)
• External service flakiness (a staging API is intermittently down)
• Rate limiting (Docker Hub pulls, GitHub API calls)

Why This Distinction Matters

If you have a flaky test and you respond by upgrading your CI runner, adding retries to your pipeline, or clearing caches — nothing improves. The test is still non-deterministic.

If you have a flaky pipeline and you respond by rewriting tests, quarantining test suites, or adding randomization seeds — nothing improves. The infra issue is still there.

The diagnostic approach is fundamentally different:

How to Diagnose: Track Patterns Over Time

The reliable way to tell flaky tests from flaky pipelines is to look at failure patterns across multiple runs — not just the latest failure.

Pattern 1: Same Test, Same Assertion → Flaky Test

If the same test fails intermittently with the same assertion error, you have a flaky test. The test name and error message are consistent. The infrastructure is irrelevant.

# This pattern repeats across runs:
FAIL src/utils/cache.test.ts
  ✕ should expire entries after TTL (3ms)
    Expected: undefined
    Received: "cached-value"

# Root cause: the test creates a cache entry, waits 100ms,
# and asserts it's expired. But the TTL timer sometimes
# hasn't fired yet. Classic race condition.

Pattern 2: Different Tests Fail Each Time → Flaky Pipeline

If the failing test changes between re-runs — or the failure is in a setup step, not a test step — you have a flaky pipeline. The tests themselves are fine; the environment is unstable.

# Run 1: npm ci fails with ETARGET
# Run 2: Jest fails with ENOMEM (exit code 137)
# Run 3: Passes
# Run 4: Docker build fails with network timeout
# → This is infra instability, not test flakiness

Pattern 3: Failures Cluster at Specific Times → External Dependency

If failures happen between 2-4 AM UTC (when cron jobs run), or during peak hours (when the staging API is under load), you have an external dependency problem. Neither the test nor the pipeline is flaky — a third-party system is unreliable.

What to Do About Flaky Tests

1. Quarantine immediately — move the flaky test to a separate test suite that doesn't block merges. Don't let it break CI for the whole team
2. Assign ownership — the original test author should fix it. If they've left the team, the code owner should
3. Fix the root cause — don't just increase timeouts. Common fixes:
   • Use deterministic time (jest.useFakeTimers(), freezegun)
   • Isolate test state (fresh database/store per test)
   • Remove execution-order dependencies
   • Mock external services properly
4. Track flake rate — measure how often each test flakes. A test that fails 1% of the time across 100 daily runs fails every day

What to Do About Flaky Pipelines

1. Add retries at the right level — retry the flaky step, not the entire pipeline:

   # GitHub Actions: retry a specific step
   - name: Install dependencies
     uses: nick-fields/retry@v3
     with:
       timeout_minutes: 5
       max_attempts: 3
       command: npm ci

2. Add resource monitoring — track memory and disk usage to catch OOM kills before they happen
3. Reduce external dependencies — cache Docker images, use a private npm registry, mock staging APIs in CI
4. Track pipeline reliability — measure success rate, time-to-fix, and rerun rate

The Real Cost: "Just Rerun It" as Team Culture

The most damaging outcome isn't a single flaky failure — it's when "just rerun it" becomes the team's default response to any CI failure. This creates several problems:

• Genuine failures get masked — a real bug gets re-run and happens to pass because of a different test order or timing
• Investigation stops — nobody looks into failures anymore because "it'll probably pass next time"
• The problem compounds — each undiagnosed flaky test or pipeline issue adds to the overall flake rate

The reliable way to tell these apart is tracking failure patterns across runs, not just eyeballing the latest one. This is a good use case for automated log analysis — Daxtack tags recurring failure signatures over time so you can see whether it's the same test or the same infra issue every time.