Measuring Hallucination
A hallucination is a confident claim with no support. You can't catch it by reading the answer alone — it sounds right. You catch it by checking every claim against what the model was actually given.
Groundedness is the metric
Break the answer into individual claims. For each, ask: is this supported by the source the model had? An answer where every claim traces back to the context is grounded; one confident sentence that doesn't is the hallucination — and it's usually the most fluent sentence in the paragraph.
LLM-as-a-judge, and its traps
A second model scores groundedness against the source. It scales where humans can't — but it has failure modes you must design around:
- It prefers longer, confident answers — length and tone bias. Control for it or it rewards the exact thing you're hunting.
- It agrees with itself — judging output from the same model family inflates scores. Use a different model as judge where you can.
- It drifts — the judge is also non-deterministic. Spot-check judge scores against human labels regularly, or you're trusting an unaudited grader.
Grounded in published practice on groundedness scoring and LLM-as-a-judge