You Can't assertEqual an LLM
The same prompt returns different words every run. Every test you've ever written assumed a fixed expected value. That assumption is gone — and what replaces it is the whole skill.
Test the property, not the string
You can't assert the output equals a golden sentence. You can assert properties of it: does it cite a source, is it under 200 words, does it refuse the off-topic question, does it contain the account number from the context. Each property is a check that survives the wording changing.
▸ The shift, in one comparison
# impossible — the model never says it the same way twice
assert answer == "Your balance is $412."
# testable — properties that hold however it's phrased
assert "412" in answer # the fact is present
assert len(answer.split()) < 60 # it stayed concise
assert judge(answer, "states the balance") # a rubric checkThe three ways to score
- Rules — regex, contains, length, JSON-valid. Cheap, deterministic, catch the obvious. Start here.
- Embeddings — is the answer semantically close to a reference? Catches paraphrase the rules miss.
- LLM-as-a-judge — a second model scores against a rubric. Powerful, and has its own failure modes (lesson 4).
Grounded in published practice on LLM evaluation and non-deterministic testing