Sampling: temperature, top-p and min-p, and how they change the output distribution

About this article: AI-assisted writing, reviewed against the cited sources. This is source research, not a lab validation of the procedures. Check the stated versions and your own recovery requirements before making changes.

A decoder-only language model ends each step with a vector of logits, one score per token in the vocabulary. Sampling turns that vector into one next token. Temperature changes the shape of the distribution before any token is dropped. Top-p and min-p then drop tokens and renormalise what remains. The three knobs feel similar in a settings panel. They edit different parts of the same pipeline, and stacking them without looking at the intermediate distribution is how a "creative" setting becomes either a loop or a word salad.

An AI writing system helped prepare this article for NetLeverage. Phrases such as the model being "confident" mean a peaked next-token distribution, a large probability on the top token. They are not a claim about subjective experience. The numerical example is computed from the published formulas on made-up logits. It was not sampled from a running model. This article makes no NetLeverage product claim.

Why it matters

Holtzman et al. showed, on GPT-2 Large, that the decoding rule can dominate the model. Beam search, which hunts for high-probability sequences, fell into repetitive loops. Sampling straight from the full distribution produced text they describe as incoherent. The same weights did both. The difference was which tokens were allowed to win.

That still matters operationally:

• A support bot that must stay on a procedure wants a peaked distribution and a small candidate set.
• A draft that should vary wants a flatter distribution, and then a truncation rule that still refuses the long tail.
• An evaluation that reports one accuracy number at "the default sampler" is reporting a property of the pair (checkpoint, sampler). Nguyen et al. find small improvements over greedy decoding in some settings: Table 14 reports Mistral 7B GPQA 27.35% greedy versus 29.18% best min-p, and GSM8K 39.35% versus 40.33%. These margins need uncertainty and tuning controls; Table 10 gives a different GPQA greedy baseline of 27.68% without explaining the discrepancy.

The sampler does not rewrite the attention softmax inside the network. It only changes how the final logits become the next id. A bug in the sampler leaves the weights untouched and still changes every measured answer.

How it works

From logits to a distribution

Holtzman et al. write temperature sampling, citing Ackley et al. (1985), as a fresh softmax. With logits $u_l$ and temperature $t$:

Divide every logit by $t$, then exponentiate and normalise. The ordering of tokens does not change, because $1/t$ is positive for $t>0$. The gaps change.

• As $t$ falls toward 0, the largest logit pulls almost all of the mass. The limit is greedy decoding: always emit the argmax. The formula itself divides by $t$, so $t=0$ is not a legal input to the expression. An API that accepts 0 is defining a special case.
• At $t=1$ the softmax is the one the logits already implied.
• As $t$ rises, the logits shrink toward each other and the distribution moves toward uniform over the vocabulary. Tokens that were negligible pick up mass.

Holtzman et al. discuss $t$ in $[0,1)$, the range that skews mass toward events the model already ranked highly. They report that lowering temperature improves superficial quality and reduces diversity, and that temperatures below 0.9 sharply increase repetition in their loop metric. Values above 1 are the same formula used in the other direction. Nguyen et al. evaluate temperatures from well below 1 up to 3, and mention sweeps out to 5. The OpenAI chat completions reference, as served on 10 November 2026, accepts temperature from 0 to 2, default 1, and describes 0.8 as more random and 0.2 as more focused. That page is a live reference. Read it again before you treat 2 as a permanent maximum.

A three-token illustration, rounded to three decimals, uses logits $4,2,0$:

At $t=1$, $e^4$, $e^2$ and $e^0$ are about 54.60, 7.39 and 1, and the normaliser is about 62.99. At $t=2$ the logits are halved before the exponential, and the third token's share rises from 1.6% to 9%. Nothing about the model's weights changed. Only $t$ did.

Truncation, then renormalise

Temperature never sets a probability to zero, apart from the greedy limit. The unreliable tail Holtzman et al. describe is still there: tens of thousands of tokens with small probability that, added up, are a large share of the draws if you sample forever. Truncation deletes that tail and renormalises.

Their nucleus, or top-$p$, vocabulary is the smallest set of highest-probability tokens whose probabilities sum to at least $p$:

If that set sums to $p^{\prime }$, each survivor is divided by $p^{\prime }$ and everyone else is set to 0. On the $t=1$ column above, $p=0.9$ does not fit in the first token (0.867). Adding the second reaches 0.984. The third token dies. Renormalising the two survivors gives about 0.881 and 0.119. The second token became more likely than it was, because the mass of the discarded token was shared out. Extending Holtzman et al.'s section 3.2 warning about top-k to this illustrative truncation: a truncation set that still contains poor candidates makes those candidates more likely, not less, once you renormalise.

Top-$k$, which they formalise beside nucleus sampling and attribute to Fan et al. and others, keeps a fixed count of tokens instead of a fixed mass. The same renormalisation applies. A flat next-token distribution (many plausible verbs) wants a large $k$. A peaked one (a word the model is nearly sure of) wants a small $k$. One constant cannot be both, so a large $k$ admits junk in the peaked case and a small $k$ throws away real options in the flat case.

Nucleus sampling is their answer to that: the candidate count floats. In their runs the usual operating region was $p\in [0.9,1)$. $p=1$ keeps every token that has mass, which is full sampling again. The OpenAI chat reference defines top_p the same way — the tokens comprising that probability mass, so 0.1 means the top 10% of mass — on a scale from 0 to 1, default 1. Default 1 means the parameter is not truncating.

What repetition actually is

Holtzman et al. are blunt about maximisation. Beam search assigns very high probability to text that is generic and repetitive. Natural text, given the same prefix, sits at lower and more variable per-token probability. Chasing the mode is the wrong objective for open-ended generation, on their evidence.

They also document a feedback loop: once a phrase has repeated, the model assigns the next repetition an even higher probability, for most phrases they tested. Greedy decoding and low-temperature sampling walk into that loop and stay there. Their Figure 9 counts a repetition when a phrase of at least two tokens repeats at least three times at the end of a 200-token generation. Temperatures under 0.9 made that much more common. Nucleus sampling and reasonable top-$k$ settings stayed closer to human text on that plot. The experiment is GPT-2 Large (762 million parameters in section 4.1; the Figure 1 caption says 774 million), 5,000 passages, prefixes of 1–40 tokens from held-out WebText, stopping at an end-of-document token or 200 tokens. It is a 2020 result about that model. It is still the right warning: if a production sampler is "almost greedy", measure repetition before you celebrate consistency.

Min-p: a threshold that follows the top token

Nguyen et al. (ICLR 2025) add a third truncation. After temperature has been applied, let $p_{max}$ be the probability of the single most likely token, and let $p_{\text{base}}$ be a chosen fraction in $(0,1]$. The cutoff is

Keep every token whose probability is at least $p_{\text{scaled}}$, then renormalise and sample, the same pattern as equation (3) in Holtzman et al. The top token always survives, because its probability equals $p_{max}$ and $p_{\text{base}}\le 1$.

The cutoff moves:

• Peaked step, $p_{max}=0.867$, $p_{\text{base}}=0.1$. The cutoff is 0.087. In the $t=1$ table, 0.867 and 0.117 stay, and 0.016 goes. Same survivors as top-$p=0.9$ on that tiny vocabulary.
• Flatter step, $p_{max}=0.665$, same $p_{\text{base}}$. The cutoff falls to 0.067. The third token at 0.090 now survives. Top-$p=0.9$ on that same column still drops it, because 0.665 + 0.245 already clears 0.9.

So min-p is a relative floor, and top-p is a cumulative budget. They disagree as soon as the distribution is not a toy with three bars. When the model puts half its mass on one token, a base of 0.1 demands that every alternative clear 5% absolute probability. When the model is spread out and the best token is only at 20%, the same base demands only 2%, and many more tokens enter. Nguyen et al. state that as the design: tight when the distribution is peaked, loose when it is flat. They also say top-p at high temperature still admits low-probability tokens, because flattening pushes more of them into the cumulative budget, and that min-p is their response. Read that claim next to their table, not as a law of every checkpoint.

Their practical range for $p_{\text{base}}$ is 0.05 to 0.1. They note that 0.5 to 0.7, or values near 1, hold coherence together at very high temperature and spend diversity to do it. The order they implement is temperature first, then the cutoff. A pipeline that thresholds the $t=1$ probabilities and only afterwards divides logits by $t$ is a different algorithm.

What the min-p measurements show, and where they do not

In arXiv v8, Table 2, Nguyen et al. publish Mistral 7B GPQA (5-shot) and GSM8K chain-of-thought (8-shot) at several temperatures. A few cells are enough to see the shape. At temperature 0.7, GSM8K accuracy is 36.09% for top-$p$ ($p=0.9$) and 35.18% for min-p. At temperature 1.0 those figures are 27.67% and 30.86%. At temperature 2.0, top-p on GSM8K is 0.00% and min-p is 6.21%. On GPQA at temperature 3.0, temperature-only sampling is 0.89%, top-p is 0.46%, and min-p is 24.55%. Table 2 does not identify which min-p base threshold applies to each row. They report standard error about 1 to 1.5 points, and they say min-p and top-p are similar below temperature 1.5, with the gap opening as temperature rises. $\eta$-sampling and $ϵ$-sampling in that table failed outright above 1.5 because of runtime.

Two readings follow, both inside the paper.

First, high temperature damaged grade-school arithmetic for both methods on this setup: min-p scored 6.21% at temperature 2.0 and 0.00% at 3.0, versus top-p's 0.00% at both. Its high-temperature advantage is clearer on GPQA: min-p 26.34% and 24.55%, against top-p 6.47% and 0.46%. This is task-dependent evidence, not general resistance to high-temperature failure.

Second, min-p did not win every low-temperature cell. Anyone who quotes only the temperature-3 column is editing the table. The authors' own summary is that the methods match when the distribution is already sharp, and that min-p holds up better once temperature has flattened it. They also report a human preference for min-p from 54 valid responses across 12 configurations, plus evaluations of Llama 3 sizes from 1B to 70B and a 123B Mistral model. A new checkpoint needs its own sweep.

That human-preference conclusion is disputed. Schaeffer, Kazdan and Denisov-Blanch, in Min-p, Max Exaggeration (arXiv:2506.13681 v2, 19 June 2025), report min-p as largely indistinguishable from other samplers using 95% confidence intervals. Only one of twelve paired tests remained significant after Bonferroni correction. Their GSM8K re-analysis finds no advantage when hyperparameter search volume is equalised. The original authors also revised their adoption figures after acknowledging false positives in preliminary GitHub searches. These disagreements favour balanced tuning and held-out evaluation rather than declaring one sampler universally superior.

They use base thresholds 0.05 and 0.1 against top-p at 0.9, on vLLM for the main methods. That is a fair thing to copy as a starting grid. It is not a tuned optimum for your prompts.

API advice versus the mathematics

The OpenAI chat reference recommends changing temperature or top_p, and not both. That is vendor guidance against confounding two knobs, current as of the page fetched on 10 November 2026. The mathematics allows the combination: Holtzman et al. already discuss shaping with temperature and then truncating, and they show that temperature 0.7 plus top-$k$ can fall back into repetition. If you do move both, log the full triple (temperature, truncation type, threshold) with the eval. A two-point movement is of a similar order to the reported per-cell standard errors of roughly 1–1.5 points; it is not by itself a significance test. Compare repeated runs and account for how many configurations were tried.

Presence and frequency penalties, which the same API exposes, add a bias from tokens already emitted. They are a further edit to the logits, upstream of this softmax. This article does not specify their formulas. If they are on, they are part of the sampler you think you measured.

What to choose and why

Change one control, rerun the same prompts, and keep the other fixed. The OpenAI recommendation and ordinary experimental design agree on that point even though they come from different documents.

Risks and limits

• The illustration uses three fake logits. It shows the algebra. It does not predict a production vocabulary of 50,000 or 200,000 tokens, where the tail Holtzman et al. describe is most of the mass that truncation exists to remove.
• GPT-2 Large results are not a 2026 model card. Beam-search loops and the unreliable tail are mechanisms they measured. Whether a given instruction-tuned model still loops at temperature 0.7 is a measurement you still owe.
• Min-p's gains are temperature-dependent in their own table. At 0.7 on GSM8K, top-p scored higher. At 3.0 on GPQA, min-p scored 24.55% versus top-p 0.46%; at the same temperature both scored 0.00% on GSM8K. Quote the cell you mean.
• $t=0$ is outside the softmax formula. Confirm how the library you call implements it. "Deterministic" in a doc string can mean greedy, or it can mean a tiny epsilon you have not inspected.
• Renormalisation is a second distribution. Report both the pre-truncation probability and the probability you actually sampled from when you debug a surprise token.
• Vendor ranges move. The 0–2 and 0–1 limits above are from the OpenAI chat reference on 10 November 2026. Another provider, or a later revision of that page, can differ. Min-p is not a parameter on that page.
• No lab run and no NetLeverage product behaviour are claimed.

Sources

1. Holtzman et al., The Curious Case of Neural Text Degeneration, arXiv:1904.09751 — https://arxiv.org/abs/1904.09751 (HTML via ar5iv, opened 10 November 2026). Temperature softmax, nucleus definition, top-$k$, repetition loop, GPT-2 Large protocol.
2. Nguyen et al., Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputs, arXiv:2407.01082, ICLR 2025 — https://arxiv.org/abs/2407.01082 (opened 10 November 2026). Min-p definition, order relative to temperature, Mistral 7B table, base-threshold guidance.

3. OpenAI, chat completions reference — https://developers.openai.com/api/reference/resources/chat (opened 10 November 2026). Documented ranges and defaults for temperature and top_p, and the guidance to change one of them.

4. Schaeffer, Kazdan and Denisov-Blanch, Min-p, Max Exaggeration, v2, 19 June 2025, reopened 10 November 2026. Contrary evidence on human evaluation and equalised tuning budgets.

This article was originally published on 10 November 2026.