Guardrails, CI, and the Interview Answer

Two layers, two jobs

• Offline evals — your golden set as a CI gate. A prompt or model change runs the whole set and fails the build if faithfulness or refusal rates drop. This is regression testing.
• Runtime guardrails — checks on live traffic: block or flag an answer that leaks PII, fails a groundedness threshold, or trips a safety filter, before it reaches the user. This is production monitoring.

# runs on every prompt / model / retrieval change
results = run_golden_set(feature, golden_set)
assert results.faithfulness >= 0.90    # regression gate
assert results.refusal_rate >= 0.95    # safety didn't slip
assert results.pii_leaks == 0          # hard fail
# below threshold -> red build, same as any failing test

The interview answer, assembled

When they ask "how would you test our AI feature?", you now have a structured answer nobody else in the loop has: define correctness as properties, not strings; build a golden set with rubrics and run it as a CI gate; for RAG, score retrieval and faithfulness separately; measure hallucination with a validated judge; red-team for jailbreaks and injection; and add runtime guardrails for what slips through. Six sentences, every one earned.

Grounded in published practice on LLM guardrails and evaluation in CI