Flaky or Broken: Prove It With a Number

Measure it

pass=0; fail=0
for i in {1..50}; do
  pytest -q tests/test_checkout.py::test_guest_checkout >/dev/null 2>&1 \
    && pass=$((pass+1)) || fail=$((fail+1))
done
echo "passed $pass / failed $fail"
# (pytest-repeat does the same: pytest --count=50 -x <test>)

• 0/50 fail — the failure came from the environment. Go to lesson 5.
• 4/50 — an 8% race. Any single trace will half-lie; timing is the suspect.
• 50/50 — not flaky. Broken. Follow the trace.

Why a "small" rate is a big deal

One test failing 0.5% of the time sounds harmless. Two hundred like it fail a build most days, and the team learns to click re-run without reading.

That habit — not the race — is what kills automation. Google has published that most of its retried test failures are flakiness, not real regressions. Once a team expects that, a real red gets ignored with the rest.

sleep(2) is how frameworks die

# bad — hides the race, taxes every run forever, still fails on a slow day
time.sleep(2)
page.click("#place-order")

# good — waits exactly as long as the condition takes, and fails honestly
expect(page.locator("#cart-total")).to_have_text("$41.98")
page.click("#place-order")

The nap doesn't fix the race — it moves it. And it charges two seconds, times every copied test, times every run, forever.

Quarantine, with a paper trail

A test measured flaky gets marked, ticketed and owned — the same day. The mark keeps the build honest; the ticket keeps the coverage loss visible.

Grounded in pytest-repeat and published flaky-test research from Google and Microsoft