The Failed Test: Debugging Automation Like a Senior · Lesson 3 of 6 · Bonus module

Shrinking the Search Space

Debugging is search. Every run should cut the suspects in half — one variable at a time.

By Shahriyar · Updated

Run less

▸ Select, repeat, stop early
pytest tests/test_checkout.py::test_guest_checkout   # exactly one test
pytest -k "checkout and not slow"                     # by name pattern
pytest --lf                                           # only what failed last run
pytest -x                                             # stop at the first failure

Split the flow

A 40-step flow that fails at step 31 has forty suspects. Copy the test and stop it at step 20. Pass → the problem is in the back half. Fail → the front. Keep halving.

Bisect time

Green two weeks ago, red today, nobody knows why? Let git find the commit — with the test as the judge:

▸ git bisect with pytest as the oracle
git bisect start HEAD <last-known-good-sha>
git bisect run pytest -x -q tests/test_checkout.py
# git checks out the midpoint, runs the test, and walks
# straight to the first bad commit. Then: git bisect reset

Kill the parallelism

Fails in parallel but passes serially (-n 0 for pytest-xdist, -p no:randomly to pin order)? Two tests share something they shouldn't — a record, a user, a port.

Grounded in the pytest docs (test selection, --lf) and git bisect

All lessons in The Failed Test: Debugging Automation Like a Senior

  1. The Four Questions That Sort Any Failure
  2. Reading the Traceback Like a Senior
  3. Shrinking the Search Space
  4. Flaky or Broken: Prove It With a Number
  5. It Only Fails in CI
  6. The Write-Up: Product Bug or Test Bug