DevOps

Why Rerunning Your Pipeline "Fixes" It (And Why That Should Scare You)

"Just rerun it" is the most expensive debugging strategy in CI/CD. It masks non-determinism, kills investigation culture, and quietly accumulates technical debt your team won't notice for months.

D

Daxtack Team

Engineering

7 min read

Your CI pipeline fails. You click "Re-run." It passes. Problem solved?

No. You just swept a non-deterministic failure under the rug. And if this happens regularly — if your team treats re-running as the fix — you're accumulating invisible technical debt that will cost you far more than the 5 minutes you saved by not investigating.

The Uncomfortable Truth

"Just rerun it" means your pipeline has non-determinism somewhere. That non-determinism has a source, and that source will keep producing failures. The only question is whether it's a small annoyance now or a major problem later.

Here's what a successful rerun is actually telling you:

  • Resource limits were hit — your job is close to the memory or disk limit. It fails when resource contention is high, passes when it's low. As your codebase grows, it will fail more often
  • A race condition exists — in your tests, in your build process, or in your deployment. Race conditions get worse under load, not better
  • An external dependency is unreliable — a package registry, a staging API, a cloud service. You're one outage from blocking every PR on your team
  • Your cache is doing something wrong — the first run after a cache miss behaves differently from subsequent runs. This is a ticking time bomb

None of these problems go away on their own. All of them get worse over time.

What It Actually Costs

Let's do the math on a team of 10 engineers:

  • Average re-run rate: 15% of CI runs need at least one rerun
  • Average PRs per engineer per day: 2
  • That's 20 CI runs per day, 3 of which need re-runs
  • Each re-run takes 8 minutes (pipeline time) + 2 minutes (context switch) = 10 minutes
  • 3 re-runs × 10 minutes = 30 minutes per day, 2.5 hours per week, 130 hours per year

At $100/hour fully-loaded engineer cost, that's $13,000/year in re-runs alone. And that doesn't count:

  • The time other engineers wait for the re-run to pass before their PR can merge
  • The merge conflicts that accumulate while PRs wait in the queue
  • The genuine bugs that get missed because "eh, it'll pass on rerun"

The Team-Culture Problem

The bigger cost is cultural, not financial. When re-running becomes normal:

1. Investigation Dies

Nobody looks at failures anymore. Why spend 10 minutes investigating when you can re-run in 2 clicks? But that 10 minutes of investigation would fix the root cause once, permanently. Instead, you pay the re-run tax forever.

2. Signal Gets Lost in Noise

When 15% of builds fail randomly, engineers stop treating failures as signal. A real regression gets merged because "that test is always flaky, just re-run." This is how bugs reach production.

3. New Engineers Learn the Wrong Habits

A new engineer joins your team. Their first CI failure, they ask: "What does this error mean?" Someone replies: "Oh, just re-run it." That engineer will never develop the debugging instinct. They'll re-run for the rest of their tenure.

What to Actually Do Instead

Track Rerun Rate as a Metric

You can't fix what you don't measure. Start tracking:

  • Overall rerun rate — what percentage of CI runs are re-runs? Healthy: under 5%. Concerning: 10-15%. Urgent: above 20%
  • Per-job rerun rate — which specific job gets rerun most often? This identifies your worst offenders
  • Time trend — is the rerun rate going up or down? An upward trend means the problem is compounding
# Quick and dirty: count re-runs in GitHub Actions
# via the GitHub API
gh run list --limit 100 --json conclusion,attempt | \
  jq '[.[] | select(.attempt > 1)] | length'

Treat Repeated Reruns as a P2 Bug

If the same job has been re-run more than 3 times in a week, create a ticket. Assign it. Set a deadline. Treat it the same way you'd treat a production bug that happens 3 times — because it's costing you just as much in developer time.

Enforce "Investigate First" Culture

Simple team rule: before clicking "Re-run," spend 2 minutes looking at the failure. Just 2 minutes. Read the error message. Check if you've seen it before. If it's a known flaky issue, note it and re-run. If it's new, investigate.

2 minutes of investigation × 3 daily re-runs = 6 minutes per day. That's nothing. And it will catch genuine regressions that would otherwise get re-run past.

Add Automated Failure Classification

The fastest way to decide "investigate or re-run" is to have failures automatically classified:

  • Is this a known flaky test? → Re-run is acceptable, but track the flake for future fixing
  • Is this a new error we've never seen? → Investigate, don't re-run
  • Is this an infrastructure issue (network, OOM)? → Re-run, but track the infra issue

If you're not already tracking which jobs get rerun most often, that's the single highest-signal metric for pipeline health. It's also exactly the kind of pattern Daxtack surfaces automatically across a project's log history instead of you noticing it anecdotally six months later.

CI/CDDevOpsPipeline ReliabilityTechnical DebtEngineering CultureFlaky BuildsBest Practices

Debug CI/CD failures in 30 seconds

Daxtack uses AI to automatically analyze your build logs, find the root cause, and suggest fixes — right in your pull request.

Related Articles