Groundedness and citations

Supported is not the same as true

Groundedness asks whether each claim in an answer can be traced to the context. It does not ask whether the context is correct — that is a separate question about the corpus. Splitting the two makes both measurable.

This matters because a fluent answer can be wrong in two different ways. It can contradict the documents, which is a grounding failure you can detect. Or it can faithfully repeat a wrong document, which is a corpus failure that no prompt will fix.

A worked case: the answer says refunds take five business days and are processed in Dublin. The context supports the first sentence and says nothing about Dublin. Groundedness is one of two claims, or 0.5. The score is not a judgement about whether Dublin is true; it is a judgement about whether the answer stayed inside what it was given.

Break the answer into claims

A sentence is a reasonable first approximation of a claim. Split the answer on sentence boundaries, then check each one against the passages it cites, or against the whole context when it cites nothing.

The check can be lexical — does the sentence’s distinctive vocabulary appear in the context — or model-based, where a judge is asked whether the passage entails the claim. The first is cheap and noisy; the second is better and costs a call per claim.

Sentence splitting is less reliable than it looks. “The starter plan is free, e.g. for one project” splits at “e.g.”, and a list separated by semicolons becomes one long claim. Merge fragments shorter than a few words into their neighbour rather than trusting a regex alone. A claim that spans two sentences is the harder case, and it is usually better treated as two claims that must each be supported.

Faithfulness as a metric

from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy

result = evaluate(
    dataset=dataset,               # question, answer, contexts
    metrics=[faithfulness, answer_relevancy],
)
print(result["faithfulness"])      # claims supported / claims made

Citations have to resolve

A citation is a promise that the passage supports the sentence next to it. Two things can break that promise: a marker that points at no passage, and a marker that points at a real passage which does not contain the claim.

The first is a formatting bug and trivial to check. The second is the interesting one, and it is why citation checking belongs in the evaluation harness rather than only in a UI.

A worked citation failure: the answer says “support tickets are resolved within five business days [1]”, and marker one points at the refund policy. The marker resolves, so the formatting check passes. A word-overlap check also passes, because the two sentences share most of their words. Only a check that asks whether the passage supports the claim catches it. That is the difference between validating markers and validating citations.

Faithfulness is not answer quality

A faithful answer can still be bad. It can be incomplete, answer a different question, or repeat a passage that is irrelevant to the user. Faithfulness measures the link to the context, not the link to the question.

Measure the second link separately, with answer relevancy or a judge that reads the question and the answer without the context. Reporting one number for both hides which failure you have: a grounded answer to the wrong question and an ungrounded answer to the right one need different fixes.

What a good score looks like

A faithfulness score near one is achievable and expected for a short answer over clean documents. A score that never leaves the middle usually means the answers are padded with general knowledge the passages do not contain.

Read the unsupported claims. They are the model’s assumptions written down: a plausible cause, an invented number, a policy that sounds like the others. That list is one of the most useful artefacts an evaluation produces.

Report the unsupported claims, not only the rate. A score of 0.8 over ten claims means two sentences to read, and those two sentences are usually a specific habit: an invented cause, a number rounded, a policy extended by analogy. Naming the habit is what lets you change the prompt.