Flaky Checks and Retry Strategies in CI

A flaky check fails sometimes and passes other times with no code change. Flakes are corrosive: they erode trust in the pipeline, they let real failures hide behind "oh, it's just a flake," and they silently raise everyone's noise floor. Retrying is a useful tool, but only the beginning. This post covers how to recognize flakes, retry them safely, and find the root causes so checks become genuinely reliable.

Why Pipelines Flake

Most flakes trace to one of a few predictable sources:

• **Timing**: a test waited too short a time for an asynchronous operation.
• **Shared state**: parallel tests touching the same database, files, or ports collide.
• **Environment drift**: a runner dependency changed between runs.
• **Uncontrolled inputs**: a test depends on current time, random data, or network.
• **Resource contention**: a test needs more CPU or memory than the runner has available at that moment.

Each flake is a signal about your test suite or your runner. Treat the symptom (retry) while you hunt the cause, but never stop at the symptom.

When Retrying Is the Right Tool

Retrying is legitimate for genuinely transient failures — a network timeout during a package download, a short-lived service restart. For those, automate the retry. GitHub Actions and most platforms let you declare retries on individual steps.

      - name: Install dependencies
        uses: nick-fields/retry@v3
        with:
          retry_wait_seconds: 15
          max_attempts: 3
          retry_on: error
          command: npm ci

That retries transient network errors on dependency installs, which are exactly the kind of flake that is almost always environmental. The key is that retry_on: error re-runs the whole step, so it protects against transient conditions and not against a genuinely broken install that will fail the same way every time.

Retrying Whole Jobs Is a Crutch

Platforms offer a "rerun failed jobs" button. It is convenient, and occasional manual use is fine. But if you find yourself rerunning failed jobs daily, you have a flake problem, not a retry problem. Reruns hide when normal operations drift out of order. Reserve manual reruns for runs you genuinely believe were environmental, and immediately open an investigation for anything that fails more than a couple of times.

Reproducing a Flake

To fix a flake you first have to catch it reliably. Run the suspect test many times in a loop and collect the failure rate and the failure mode.

# run a specific flaky test 100 times to reproduce it
for i in $(seq 1 100); do
  npm test -- --grep "token refresh" || echo "FAILED at iteration $i"
done

Seeing the failure rate tells you whether it is rare and timing-related (low rate) or systemic (high rate). Capture the full stack trace and the environment, then look for the categories above. Timing assumptions — fixed sleeps instead of waits, hardcoded timeouts — are the single most common fixable cause.

Making Checks Reliable, Not Retried

Go after root causes rather than piling on retries:

• Replace fixed sleeps with proper waits for the condition you actually need.
• Isolate tests that touch databases, files, or ports into serial groups.
• Give tests deterministic inputs: freeze clocks, seed random, mock the network.
• Size runners so heavy tests do not starve for CPU.
• Promote a flaky test to its own job so it does not tangle with others.

A flake you can reproduce on demand is a flake you can fix. A flake you only see occasionally in the wild is a flake you have never captured.

Quarantine as a Last Resort

When a test is flaky and you cannot fix it immediately, quarantine it: mark it to be skipped in the gate but tracked elsewhere, with a ticket and an owner. This keeps the pipeline honest — you do not hide flakes behind retries — while removing a nondeterministic red light. The danger is quarantines that never get fixed, so require an owner and a deadline for every quarantined test.

Takeaway

Retry only what is genuinely transient, reproduce flakes before fixing them, and remove nondeterminism through better tests and better runners. Reliability comes from eliminating the cause, not from clicking rerun.

For consistent test timing and enough cores so flakes vanish, a Netbay VPS sized to your suite is a dependable runner — spin one up at netbayhosts.in.