Evaluating LLM systems: test sets, regression gates and statistical significance

About this article: AI-assisted writing, reviewed against the cited sources. This is source research, not a lab validation of the procedures. Check the stated versions and your own recovery requirements before making changes.

An LLM system is the model plus the prompt, the tools, the retrieval, and the decoder settings. A test set is a frozen list of inputs with a rule for what a good output is. A regression gate fails the build when an agreed measurement moves the wrong way. A significance test asks whether that movement is distinguishable from sampling noise. Those are three different decisions, and a leaderboard number does not make any of them.

An AI writing system produced this article for NetLeverage's editorial studio. Statements about what a model "prefers" when it judges another model are reports of the cited experiments, not claims about experience. No evaluation harness was run for this article. Figures below are the papers' own, on the models they named.

Why it matters

A single accuracy on a public exam rewards whatever the exam happens to score. HELM's complaint, in Liang et al., is that earlier benchmarks were collections of datasets, each with one canonical metric, usually accuracy. The relative importance of accuracy, robustness, fairness, and cost depends on the use. A mobile assistant and a batch classifier do not share a utility function, so one scalar hides the trade.

The other failure is a release that "got better" because the test set was small, the judge liked longer answers, or the new model had seen the questions. Dror et al. put the statistical point in one line: if you validate a hypothesis from an experiment, you need to know the result is not a coincidence. Zheng et al. show a judge model can flip its verdict when the same two answers swap places. A gate that ignores either paper will bless noise and bias.

How it works

A scenario is not a score

HELM defines a scenario as a triple: task, domain, and language. The paper's core set is 16 scenarios and 7 metric categories: accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency. The authors measured 98 of the 112 possible core scenario/metric pairs, or 87.5%, deliberately keeping non-accuracy measures in the core scenarios. Another 7 targeted evaluations use 26 further scenarios, including memorisation, copyright, and disinformation. The taxonomy is also a list of what they did not cover. They name question answering for neglected English varieties, and they say the concrete selection is English-only apart from some English varieties such as African American English and national varieties.

That structure is the part to copy, not the 2022 model ranking. Three rules in the paper are operational:

1. Name the missing cells. A holistic eval states the scenarios and metrics it aspires to and then says which ones it actually ran.
2. Hold the adaptation fixed. HELM compared models with the same few-shot prompts, described as simple and generic, so the comparison is of models under one interface. The authors note that chain-of-thought, prompt decomposition, or prompt tuning could change the ranking, and they leave that axis for later. They also note that a model weak under their prompts may be strong under others.
3. Treat contamination as unknown unless you measured it. The paper says several commercial models do not disclose training data, and that exposure to the evaluation instances is a real possibility they only partly understand. A rising score on a public set is compatible with memorisation.

Their study ran 4,939 evaluations, about 12.2 billion tokens, about 19,500 GPU hours on the open models, and 38,001 US dollars in API fees. A product suite should be smaller, repeated, and still explicit about the cells it leaves empty.

Behavioural tests catch what an average hides

Ribeiro et al. (CheckList) treat evaluation the way a software team treats unit tests. Capabilities are rows. Three test types are columns.

Their sentiment examples are templates filled from lexicons. Negation is an MFT: "I {NEGATION} {POS_VERB} the {THING}." Changing a location or a person name is an INV for sentiment, because the sentiment should survive the swap. Appending "You are lame." is a DIR: sentiment should not become more positive. They also test typos by swapping neighbouring characters, and neutral URL or handle insertions that should not move the label.

For illustration, a 92 percent aggregate can coexist with a capability that fails half the time. An average over easy and hard items hides the cell. CheckList's point is to report the failure rate on the cell. The paper is from 2020 and its demonstrations are task-specific models for sentiment, duplicate-question detection and machine comprehension, including three commercial sentiment services, not a 2026 chat system. The three test types do not depend on that era. For an LLM system the same grid applies to tool choice, citation faithfulness, refusal, and formatting. An INV for a retrieval answer is: swapping an irrelevant sentence in the retrieved context must not change the cited fact. A DIR for a calculator tool is: replacing 2+2 with 2+3 must change the number, and must change it in the direction the arithmetic requires. Those last two sentences are editorial applications of the paper's test types, not experiments in the paper.

A regression gate is a pre-committed rule

A gate is not a dashboard. Before the candidate runs, write down:

• the test-set version, so a passing build can be tied to a file hash;
• the model snapshot, the prompt version, the tool schema version, and the decoding settings;
• the metric and the slice;
• the fail condition.

HELM's standardisation is the reason the prompt and the few-shot count belong in the gate. If the new build uses a different prompt, you are not measuring the same system. CheckList's MFTs and INVs belong in the gate as rates. A natural fail condition for an invariance is a non-zero failure rate, or a rate you set before looking at the candidate. Setting the threshold after you see the chart is how a gate becomes a story.

Two comparisons get conflated.

Absolute gate. The candidate's slice score must clear a floor, or its invariance failure rate must stay under a ceiling. This can fail even when the candidate is statistically tied with the last release, if both sit under the floor. That is what you want for a safety cell.

Paired comparison. The candidate is scored on the same items as the last accepted build. The question is whether the difference is larger than the noise of this test set. Dror et al. define that difference as $\delta$ observed on the test set $X$ under metric $M$. The p-value is the probability, under a stated null hypothesis, of a difference at least this extreme. You reject the null only when the p-value is below a threshold $\alpha$ you chose beforehand. Their survey of ACL 2017 found the word "significant" used as a synonym for large or important. In a release note, keep the statistical sense. "The exact-match rate rose by 4 points and the paired test at $\alpha =0.01$ rejected the null" is a different sentence from "a significant improvement."

A gate can require both: the paired test shows a drop, and the drop crosses a practical floor. A statistically detectable drop of 0.1 point on a huge set may be irrelevant to users. A 10-point drop on 30 items may be obvious to users and still fail to reject the null, because the set is too small. Failing to reject a false null is a type II error. A statistically real but practically tiny effect is not a type I error; rejecting a true null is a type I error, and a properly constructed test bounds its probability by $\alpha$. Power is the probability of avoiding the type II error. They do not give a universal sample size. They do say you pick the test that respects the bound on type I error and then maximises power.

Which test matches the metric

Dror et al. split tests by what they assume.

If you know the distribution of the difference under the null, a parametric test uses that distribution and is more powerful and cheaper. If you do not, a parametric assumption can produce a wrong conclusion. Their reading of ACL 2017 is that the t-test was the most common named test, used 17 times among 42 papers naming one, which assumes independent identically distributed observations and a normal statistic. Many NLP metrics do not meet that. BLEU, ROUGE, and METEOR exist precisely because several outputs can be acceptable, so the statistic is not a simple per-item accuracy. The t-test is the wrong default for those scores.

When the distribution is unknown, they prefer sampling-based tests, which keep the metric values. A paired bootstrap resamples the set and recomputes the difference. A permutation test reassigns which system produced which output and asks how often the shuffled difference is at least as large. The sign test keeps only the direction of each item and has less power. McNemar's test is the one they highlight for paired right/wrong outcomes: a 2-by-2 table, null that both systems are correct on the same proportion of items, chi-squared with one degree of freedom when disagreements are plentiful. Use it for "right or wrong on these questions." With few disagreements, use the exact binomial form rather than the large-sample chi-squared approximation. Do not use McNemar for a 1-to-5 score or a BLEU delta. Other paired procedures require their own scale and distribution assumptions.

All these standard tests depend on assumptions about independent observations, a limitation Dror highlights for language data. Cases generated from one template or drawn from one customer may be dependent. Choose a suitable clustered analysis or disclose that limitation; treating dependent rows as independent can overstate precision. The paper also warns that bootstrap tests are less effective on small sets and resampling can be costly on very large ones.

Multiple slices multiply the chance of a false claim. Dror et al. report that of 110 ACL 2017 papers using multiple datasets, 3 corrected for multiplicity, all with Bonferroni. In their TACL sample, 4 of 19 papers that should have corrected did so. Bonferroni divides $\alpha$ by the number of comparisons. It is conservative and it is the correction they actually observed in use. Their own proposal for dependent folds, replicability analysis with a K-Bonferroni estimator, is in the cited 2017 companion paper and is more machinery than a first gate needs. What the first gate needs is a pre-declared list of slices, not a fishing trip across every customer and every prompt variant after the run.

Dror et al. flag dependence and possible overlap among fold test sets. Their proposal is a p-value per fold followed by dependent-dataset replicability analysis using K-Bonferroni, with the required number of rejections fixed in advance. That is more specific than applying a generic correction to fold means. For an LLM regression suite the cleaner design is one frozen test set, not a rotated split of the same pool, and a paired comparison on that set.

When the metric is another model

Zheng et al. study using a strong LLM as a judge of chat answers, against human votes. On MT-bench and a sample of Chatbot Arena votes, GPT-4's agreement with humans exceeded 80 percent, and on MT-bench the non-tie agreement with humans reached 85 percent, against 81 percent agreement among the human experts in that study. They describe that as matching human-human agreement in those datasets. It is not a certificate that a judge you prompt next year, on your tickets, will match your users.

The biases they measured are the reason a judge is a poor sole gate:

• Position. On pairs of similar answers, the judges preferred the first position often enough to matter. GPT-4's consistency across a swap was 65.0 percent on their default-prompt stress test. The fix they recommend is to judge both orders and count a win only if the same answer wins twice. Inconsistent swaps are ties. A few-shot judge raised that consistency to 77.5 percent at about four times the prompt cost, and they were not sure whether few-shot examples would introduce new biases, so the default stayed zero-shot. The authors also warn that high consistency need not mean high accuracy.
• Verbosity. They built a "repetitive list" attack that restates a list without new information. Judges, including GPT-4 to a lesser degree, preferred the longer answer. A gate on "helpfulness" that does not control length will accept padding.
• Self-preference. GPT-4's win rate for itself was about 10 points higher than the human win rate in their figure; Claude-v1's was about 25 points. The authors say the data cannot establish a self-enhancement bias, given limited data and small differences. Separately, they could not isolate style by rephrasing without potentially changing quality. The practical rule survives the caution: do not let the only judge be the model that produced the candidate, and do not treat a higher self-win as a quality gain.
• Reasoning the judge can do, and still miss. On elementary maths, GPT-4 could solve the problem when asked directly and still mark a wrong answer correct when that answer sat in the prompt. A reference-guided judge, which first answers independently and then grades against that answer, cut their reported failure rate on a small maths set from 14 of 20 judgements to 3 of 20 (70% to 15%), using 10 questions with positions swapped. Chain-of-thought grading was not enough on its own, because the judge repeated the candidate's mistake. For multi-turn, showing the full conversation in one prompt beat splitting turns, which caused the judge to lose track of which utterance it was grading.

A regression gate that uses a judge should therefore swap positions, cap or measure length, keep a reference answer for anything with a checkable result, and keep a human-labelled slice whose agreement you recompute when you change the judge prompt. Tool-use correctness and arithmetic are checkable without a judge. Prefer the checker.

What to choose and why

Temperature and seeds belong in the recorded system. HELM fixed the prompt. If you sample, the gate's unit is either a greedy or temperature-zero decode, or a mean over a pre-set number of samples with a paired comparison that accounts for that variance. Adding samples until the p-value looks nice is another way to choose $\alpha$ after seeing the data.

Our recommendation: version the set, the prompt, and the judge prompt in the same change as the code. A green build from an edited test is not a regression result.

Risks and limits

• HELM's 16 scenarios and 7 metrics are a 2022 selection. The method is the taxonomy and the fixed adaptation. The rankings are not a current model chart.
• CheckList demonstrations include sentiment, duplicate-question detection and machine comprehension on 2020 models. Applying INV and DIR to tools and citations is an editorial extension.
• Zheng et al.'s agreement numbers are for GPT-4 as a judge, on MT-bench and Arena votes, against the humans they recruited. GPT-3.5 and Claude-v1 were less consistent under order swaps (46.2% and 23.8%) than GPT-4 (65.0%). Re-measure agreement when the judge changes.
• Dror et al. do not bless a particular $\alpha$. They insist you fix it first. Bonferroni controls the family-wise error and will make some real drops look quiet. That is the trade for fewer false alarms.
• A non-significant p-value is not evidence that two systems are equal. It is a failure to reject the null on this set.
• Public test sets leak into training corpora. HELM states that limit directly.
• No harness was executed for this article, and this article claims no NetLeverage product behaviour.

Sources

Sources checked on 10 November 2026; access limitations are identified below.

1. Liang et al., Holistic Evaluation of Language Models, arXiv:2211.09110 — https://arxiv.org/abs/2211.09110 (HTML opened). Scenarios as (task, domain, language), 16 core scenarios, 7 metric categories, standardised few-shot adaptation, contamination caveat.
2. Ribeiro et al., Beyond Accuracy: Behavioral Testing of NLP Models with CheckList, arXiv:2005.04118 — https://arxiv.org/abs/2005.04118 (HTML opened). MFT, INV, and DIR.
3. Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, arXiv:2306.05685 — https://arxiv.org/abs/2306.05685 (HTML opened). Position, verbosity, and self-win measurements; swap protocol; reference-guided grading; agreement figures.
4. Dror, Baumer, Shlomov, and Reichart, The Hitchhiker's Guide to Testing Statistical Significance in Natural Language Processing, ACL 2018 — https://aclanthology.org/P18-1128/. p-values, type I and II error, McNemar, bootstrap and permutation tests, multiplicity.

This article was originally published on 10 November 2026.