Critique and Prune — Where AI-Generated Tests Go Blind

The idea, in one line

Treat every AI batch as a rough draft, not a finished answer. Your value is knowing, from memory, the exact places AI reliably goes blind — and filling those gaps yourself.

The four blind spots AI keeps missing

AI drafts strong happy-path and simple-negative cases, then skips the ones that catch real production bugs. Learn these four by heart:

• Locale — non-English names, right-to-left text, a comma used as the decimal point (1,50 instead of 1.50).
• Timezone — clocks changing for daylight saving, events crossing midnight, and the difference between UTC and a user's local time.
• Multi-tenancy — when many customers share one system, can customer A somehow see customer B's data?
• Deep negatives — a reused one-time token, a tampered request, or a payment that fails halfway through.

See it work

One practical move: tag each generated case with the angle it claims to cover, then check the batch against a required list. Anything missing is a gap a human must fill.

# The angles every batch should cover.
REQUIRED = {"happy", "boundary", "negative",
            "security", "locale", "timezone", "multi_tenant"}

def coverage_gaps(cases):
    covered = {c["angle"] for c in cases}
    return REQUIRED - covered   # what's still missing

def test_ai_batch_has_no_blind_spots():
    ai_cases = load_generated_cases("reset_password.json")
    gaps = coverage_gaps(ai_cases)
    # Fail loudly so a human fills the gaps in
    assert not gaps, f"AI missed angles: {sorted(gaps)}"

Read it top to bottom: you list the angles you expect, subtract the angles the AI actually covered, and whatever is left over is your to-do list.

Advanced — the gap list is your story

The Thoughtworks study found roughly a quarter of generated cases were ambiguous, and the harder, non-functional angles were skipped unless demanded. So write down what the AI missed. That gap list is the concrete proof of your judgment — and it's the exact story you tell an interviewer about how you review AI work.

Grounded in Thoughtworks' study on AI-generated test cases and industry guidance on validating them