Parallel, Retries & Flaky-Test Quarantine

The idea, in one line

Run tests at the same time to go faster, and quarantine flaky tests instead of hiding them. Speed and honesty, kept separate.

Running in parallel

Parallel runs come from pytest-xdist. -n auto starts one worker per CPU core; -n 4 pins the count. Because workers run tests in any order, this only works if your tests are independent — no shared mutable state, no assumptions about order. If some tests must stay together, --dist loadscope groups them by module or class per worker.

Retrying the right way

Retries come from pytest-rerunfailures. --reruns 2 re-runs a failure, --reruns-delay 1 waits a second between attempts, and --only-rerun limits retries to specific errors — retry a timeout, never an AssertionError. Retries done loosely hide real bugs, so scope them tightly.

# Fast: one worker per core; keep related tests together
# pytest -n auto --dist loadscope

# Tight retries: only network flakiness, never assertions
# pytest --reruns 2 --reruns-delay 1 --only-rerun TimeoutError

import pytest

# Quarantine a known-flaky test out of the blocking suite
@pytest.mark.quarantine
@pytest.mark.flaky(reruns=2, reruns_delay=1)
def test_live_search_suggestions():
    ...

# In CI, the main job skips quarantine; a separate job runs it:
#   pytest -m "not quarantine"   # blocking
#   pytest -m "quarantine"       # non-blocking, reported only

Read it top to bottom: the flaky test is tagged so the main build skips it, and a separate non-blocking job runs it and reports the result without failing everyone.

Advanced — quarantine with a deadline

When a test goes flaky, don't delete it and don't let it fail the build for everyone. Mark it, run it in a separate non-blocking job, and file a ticket to fix the root cause. Quarantine is a holding pen with a deadline, not a graveyard — track what's in it or the list grows forever.

Grounded in the pytest-xdist and pytest-rerunfailures documentation