Evaluating RAG: retrieval metrics, faithfulness and building a test set

About this article: AI-assisted writing, reviewed against the cited sources. This is source research, not a lab validation of the procedures. Check the stated versions and your own recovery requirements before making changes.

A retrieval-augmented system fails in two places, and one number cannot see both. Either the right chunk never reached the generator, or the generator wrote something the chunks do not support. Precision, recall and normalised discounted cumulative gain (nDCG) measure the first failure, and they need a human judgement of which chunks matter. Faithfulness, in the sense used by the Ragas paper, measures the second, and it does not check whether the chunk itself is true.

An AI writing system produced this article for NetLeverage's editorial studio. Nothing here is a report of running a judge model, a search stack, or a labelling exercise. Figures are the ones printed in the sources. No NetLeverage product is described.

Why it matters

Teams collapse three questions into "the RAG score":

1. Did we fetch a chunk that contains the answer?
2. Did the answer stick to the chunks we fetched?
3. Is the answer actually true, and is it the answer the user asked for?

Optimising (2) alone rewards a model that copies a wrong chunk verbatim. Optimising a "focused context" score alone rewards a short context that dropped the one sentence the user needed. Tuning chunk size, the fusion window and the prompt on the same questions you then publish overstates the result. Manning, Raghavan and Schütze state that last point for classical search: parameters tuned to maximise a score on a collection should be reported on a different collection, or the number describes the tuning set, not the next questions.

How it works

The retrieval counts

Chapter 8 of Introduction to Information Retrieval (Cambridge University Press pre-publication draft, 17 November 2007) defines the set measures for documents; this article applies the same units to retrieved chunks. Let a retrieved chunk be a true positive if a judge marked it relevant, a false positive if it was retrieved and marked non-relevant, and a false negative if it was relevant and not retrieved. Then

Precision is the fraction of returned chunks that were relevant. Recall is the fraction of relevant chunks you returned. $F_1$ is their harmonic mean. The book prefers the harmonic mean because returning the entire collection yields recall 1 and a precision near zero, and an arithmetic mean of those two still looks respectable. The harmonic mean stays near the worse of the two.

Those three are properties of an unordered set. A generator does not see an unordered set. It sees the first $k$ chunks, in order. Precision at $k$ and recall at $k$ cut the list there. For a system that puts ten chunks in the prompt, recall at 10 is the retrieval number that matches the product. Recall at 100 flatters a retriever whose good hit sits where the model will never read it.

Mean average precision (MAP) compresses the whole ranked list. For one information need, average precision is the mean of the precision values computed at each rank where a relevant chunk appears. A relevant chunk that never appears contributes 0. MAP is the arithmetic mean of those per-query values, so a query with one relevant chunk counts as much as a query with fifty. The book notes that MAP on a single system often swings widely across queries, for example between 0.1 and 0.7, and that queries must be numerous and varied before the average means anything. There is often more agreement on one query across systems than on one system across queries.

nDCG is the graded metric. For query $j$, with assessor score $R(j,m)$ on the chunk at rank $m$, the book gives

Here the outer average runs over queries and $Z_{kj}$ normalises each query's ideal ranking to 1. Base-2 logarithms are used for clarity; a consistent log base cancels in this normalisation. Define a policy separately for queries with no judged relevant items. If fewer than $k$ chunks came back, the sum stops early. With binary labels, $2^1-1=1$ and $2^0-1=0$, so a relevant chunk contributes $1/{\log }_2(1+m)$ and a non-relevant chunk contributes nothing. Rank 1 is divided by ${\log }_{2}2=1$. Rank 2 is divided by about 1.58. A relevant chunk at rank 10 is divided by ${\log }_{2}11\approx 3.46$. Moving the right chunk from position 1 to position 10 is not a small cosmetic change in this formula. Graded labels, which the book attributes to work by Järvelin and Kekäläinen, let "useful background" count less than "this chunk answers it" without pretending the background is irrelevant.

Two limits sit inside the definitions.

You cannot compute recall without a relevant set. Precision at $k$ only looks at what you returned. Recall asks what you missed. If nobody has marked the chunks that contain the answer, recall is not a number. It is a guess.

Pooled judgements do not cover the corpus. For a large collection the book says you do not judge every chunk. The usual method is pooling: judges label the top of the lists from several systems, plus maybe a Boolean search or an expert's finds. A chunk that no system in the pool ever surfaced has no label. A later retriever that finds a genuinely useful chunk outside that pool is holding an unjudged item. Calling it irrelevant is an evaluation convention, not a newly obtained judgement. Incomplete pools can bias comparisons in either direction depending on what is omitted. The honest record is "unjudged", kept separate from "judged non-relevant".

Judges disagree. The book treats that as normal, not as a broken gold standard, and it gives Cohen's kappa,

where $P(A)$ is the fraction of items on which judges agreed and $P(E)$ is the agreement expected from their label rates. Kappa is 1 for perfect agreement, 0 for chance, and negative for worse than chance. The chapter's rule of thumb, with the caveat that cutoffs depend on the use: above 0.8 is good, 0.67 to 0.8 is fair, and below 0.67 is a dubious basis for an evaluation. Agreement on binary relevance in TREC and in medical collections, it says, usually lands in the fair band. Swapping which judge is treated as gold can move the absolute score a lot and, in the experiments the chapter summarises, usually does little to the ranking of systems. That general observation does not guarantee a stable ranking on your test. The chapter also cites work where the relevance measure substantially changes rankings with graded judgements. Our recommendation is to assess sensitivity to adjudication and grading choices before claiming a small improvement.

The same section warns that relevance of one chunk is scored as if it were independent of the others. A second copy of the same paragraph is relevant and almost useless. Carbonell and Goldstein's marginal relevance, cited there, asks whether the chunk still adds something after the reader has seen the rest. Duplicate chunks from overlapping windows will inflate precision and nDCG if each copy is labelled relevant. The metric is then grading redundancy.

Information needs, the book says, should be written by people who know the domain and should look like real use. Random bags of terms do not. And the parameters — here, chunk length, overlap, $k$, fusion constant, prompt — get tuned on a development collection and frozen before the test collection is scored.

Faithfulness is support, not truth

Es, James, Espinosa-Anke and Schockaert define Ragas for the case where you do not have a reference answer. A question $q$ yields context $c(q)$ and system answer $a_s(q)$. They score three things with gpt-3.5-turbo-16k, and none of the three is "true in the world".

Faithfulness. The answer is faithful if its claims can be inferred from the context. An LLM splits the answer into statements $S$. A second prompt asks, statement by statement, whether the context supports it, with a yes or no after a short explanation. If $V$ is the supported subset,

A sentence that copies the chunk scores as supported even when the chunk is outdated or wrong. A sentence that adds a true fact from the model's parameters scores as unsupported. The metric punishes the second and ignores the first. That is the right penalty if the product promise is "we only say what the retrieved text says". It is the wrong penalty if the promise is "we are correct".

Answer relevance. Factuality is explicitly not part of this score. The judge writes $n$ questions that the answer could be the answer to, embeds them with text-embedding-ada-002, and averages the cosine with the original question. The paper says the score penalises an incomplete answer and a redundant one. It does not say the answer is right. The paper writes $n$ as a parameter and does not, in the section that defines the metric, fix a number. Do not invent one when you reimplement it.

Context relevance. The judge extracts sentences from $c(q)$ that help answer $q$, or returns "Insufficient Information". The score is

This falls when the context contains sentences the judge does not need. It does not rise because a gold chunk was found, and it does not fall because a necessary fact was never retrieved. A one-sentence context that happens to be on topic scores 1. A longer context that contains the answer plus the surrounding policy scores lower. Using CR as a recall target will push you toward short contexts. Recall of labelled chunks is a different measurement and still has to be done.

Their human check is WikiEval: 50 Wikipedia pages on events from the start of 2022, questions and answers drafted with ChatGPT from the introduction, then pairwise contrasts. Faithfulness compared an answer written with the page against one written with no context. Answer relevance compared a normal answer with one prompted to be incomplete. Context relevance padded the page with back-link text. Two annotators agreed on about 95% of faithfulness and context-relevance pairs and about 90% of answer-relevance pairs. That is raw agreement, not kappa. Ragas matched the human preference on 0.95, 0.78 and 0.70 of pairs. A 0–10 score from the same model matched on 0.72, 0.52 and 0.63, and a direct "pick the winner" prompt on 0.54, 0.40 and 0.52. The authors call context relevance the hardest, especially on long contexts. The paper's own low-faithfulness example swaps Christopher Nolan and Cillian Murphy for James Cameron and Tom Cruise. That contrast is blunter than a half-supported sentence in a policy answer. WikiEval shows the procedure can track this kind of pair. It does not calibrate the metric on another corpus.

What ARES changes, and what it still needs from a person

Saad-Falcon, Khattab, Potts and Zaharia argue that a fixed prompt, which is what Ragas uses, does not adapt to a new corpus and comes with no interval on the score. ARES takes three inputs: passages from the target corpus, a human validation set of about 150 annotated examples or more, and at least five in-domain question–answer examples. It then generates synthetic questions with FLAN-T5 XXL, drops a question if the retriever cannot return the source passage at rank 1, and builds negatives two ways. A weak negative is a random other passage, or a random other answer. A strong negative is another passage from the same document (or a BM25 neighbour if the document has only one passage) or an answer prompted to contradict. Separate DeBERTa-v3-Large classifiers, one per criterion, are fine-tuned to label context relevance, answer faithfulness and answer relevance.

The score reported for a system is not the raw average of those labels. Prediction-powered inference uses the human set to correct the judge and to put a confidence interval on the corrected score. In their experiments the interval is at 95%. Systems are ranked by the midpoint of the interval. The in-context baseline they compare against is gpt-3.5-turbo-16k, version 10/23, and the Ragas release they ran is 0.0.18.

On the numbers they print: across KILT and SuperGLUE sets, ARES beat that Ragas setup by 59.3 percentage points of accuracy on context relevance in the introduction (section 5.1 instead gives 59.9) and 14.4 points on answer relevance, on average. Kendall's $\tau$ averaged 0.065 higher for context relevance and 0.132 higher for answer relevance. On the AIS attribution set, their hallucination rate sat within 2.5 points of the ground-truth average. They also say the method needed 78% fewer annotations than their annotation baseline. Those are their comparisons, on their splits, with those model versions.

The splits themselves are a limit you should see before quoting the gap. For KILT and SuperGLUE they did not rank a set of production RAG stacks. They built mock systems by mixing gold triples with random or same-document negatives so that the known success rate ran from 70% to 90% in steps of 2.5 points. They state that they did not score answer faithfulness on those sets, because they had no human-annotated hallucinations there. Faithfulness is reported on AIS instead. A 59-point accuracy gap on a mock mix is evidence about judges on that mix, not a promised gain on your traffic. The paper also evaluates real retriever/generator combinations on NQ, WoW and FEVER: ranking tau averaged 0.91 for context relevance and 0.97 for answer relevance, 0.16 and 0.15 above Ragas. This real-system evidence should be reported alongside the mock-mixture limitation.

Copy the design, not the headline gap: filter synthetic questions that miss their source passage, add near-miss negatives, and keep a human set of both good and bad cases to correct the judge.

What to choose and why

Write the rules before anyone sees a score. Questions should look like real use; Manning warns that random term combinations do not. Synthetic questions can train a judge, as in ARES, after the source-passage filter. They are a weak picture of traffic if they are the whole test.

For each test question, mark chunks that suffice to answer it. Pool lexical and vector hits so the labels are not "what our current system already returns". Our labelling recommendation is to track unjudged chunks separately and report coverage; choose and disclose the scoring convention rather than silently treating unknown items as labelled. If kappa on a double-judged sample is in the chapter's dubious band (below 0.67), fix the guideline first. Overlapping copies of one passage need a rule, or precision counts the same paragraph twice.

Tune chunk size, $k$ and the prompt on the development set. Score the test set after those choices are frozen. Store the corpus snapshot, the chunker, $k$, the judge name and version, and the prompt beside the number. Ragas names gpt-3.5-turbo-16k and text-embedding-ada-002. ARES names RAGAS 0.0.18 and gpt-3.5-turbo-16k dated 10/23. A later library metric is a different definition until you have opened it. If the only judge is an LLM, keep a human set of at least about 150 good and bad cases, the minimum ARES names; its main experiments used 300, and report agreement with it.

Risks and limits

• Faithfulness is not truth, and context relevance is not recall. Both definitions are in the Ragas paper. Mixing them up produces a confident regression dashboard that moves the wrong way.
• WikiEval is 50 constructed pages and pairwise choices. High agreement with humans there, especially where the negative answer names the wrong director, does not calibrate a judge on long internal documents.
• ARES's largest accuracy gaps are on mock mixtures with known positive rates, and faithfulness was not part of the KILT and SuperGLUE ranking. Quote those conditions with the delta, and retain its real-system ranking results.
• An empty grade list is "not labelled". It is not a MAP or nDCG of zero. A chunk outside the pool stays unjudged until a person sees it.
• Judge disagreement moves absolute scores more than system order, in the textbook's summary. That is not a universal test for significance; graded judgements and different relevance measures can also change rankings.
• No labelling job was run for this article. Model names are those printed in the papers on 10 November 2026. A current API alias can differ.
• No NetLeverage product behaviour is claimed.

Sources

Claim-level notes are in the evidence file. All opened 10 November 2026.

1. Manning, Raghavan and Schütze, Introduction to Information Retrieval, chapter 8, pre-publication draft dated 17 November 2007 — https://ics.uci.edu/~djpatter/classes/2008_01_01_INF141/Materials/irbookonlinereading.pdf. The 2009 Stanford chapter could not be reopened; the article's textbook claims are scoped to this draft.
2. Es, James, Espinosa-Anke and Schockaert, Ragas: Automated Evaluation of Retrieval Augmented Generation, arXiv:2309.15217 — https://arxiv.org/html/2309.15217
3. Saad-Falcon, Khattab, Potts and Zaharia, ARES, arXiv:2311.09476 — https://arxiv.org/html/2311.09476

This article was originally published on 10 November 2026.