RLHF and DPO: what the alignment training optimises

About this article: AI-assisted writing, reviewed against the cited sources. This is source research, not a lab validation of the procedures. Check the stated versions and your own recovery requirements before making changes.

Reinforcement learning from human feedback, in the InstructGPT setup, fits a model to the rankings of a hired labelling team on a particular prompt distribution. Direct preference optimisation fits a policy to the same kind of ranking with a classification loss, and it does so by rewriting the usual reward-plus-KL objective so that the reward never appears as a separate network. Neither procedure optimises truth, safety, or "what users want" in the abstract. Each optimises a preference model estimated from the pairs it was given, pulled back toward a reference policy by a KL penalty.

An AI writing system produced this article. No policy was trained. Preference rates and dataset sizes are the papers'. Statements about what a model "wants" do not appear here; the papers measure rankings and losses. No NetLeverage model is described.

Why it matters

Ouyang and colleagues open with the point the rest of the method exists to answer: making a language model bigger does not make it follow a user's intent. Their labelers preferred a 1.3 billion parameter InstructGPT model to the 175 billion parameter GPT-3, on the authors' own prompt distribution. The models share an architecture. The difference they name is the human data. If you are about to buy a larger base model to fix instruction-following, that comparison says the objective may matter more than the parameter count, and it says so only for those labelers and those prompts.

The same paper records the cost. After RLHF, public-task scores fell on SQuAD, DROP, HellaSwag, and WMT 2015 French-to-English. They call this an alignment tax. The preference objective and the pretraining objective are not the same function. Improving one can hurt the other.

How it works

The three stages InstructGPT actually runs

The method follows Ziegler et al. and Stiennon et al. Ouyang et al. hire about 40 contractors who passed a screening test. The prompt pool is mostly English — they say over 96% — from labeler-written prompts and from prompts submitted to early InstructGPT models on the OpenAI API, with personally identifying prompts filtered. Labelers are told to infer the user's intent, skip inputs where the task is unclear, and to weigh truthfulness and harmful outputs using the instructions they were given and their own judgement. The signal represents that instructed pool, mostly English-speaking people in the United States or Southeast Asia; API customers were not a representative sample of all users. Training instructions emphasised helpfulness, while final evaluations prioritised truthfulness and harmlessness. These are conditions of the results, not a universal specification of alignment.

The stages are:

1. Supervised fine-tuning. Labelers write the output they want for a prompt. A pretrained GPT-3 is fine-tuned on those demonstrations. Their SFT training set is about 13,000 prompts. They train 16 epochs with cosine decay and residual dropout 0.2. Validation loss overfits after one epoch. Training longer still improved the reward-model score and the human preference ratings, so they select the SFT checkpoint on the reward-model score, not on the validation loss.

2. Reward model. Labelers rank between $K=4$ and $K=9$ completions of the same prompt. Every pair is a comparison, so one ranking yields $\left(\genfrac{}{}{0ex}{}{K}{2}\right)$ pairs. Those pairs are highly correlated. Shuffling them into independent rows overfit the reward model in a single pass, because each completion then drives $K-1$ updates. They put all pairs from one prompt into one batch element, which also means one forward pass per completion instead of one per pair. The loss is

where the sum covers the ordered preference pairs from one ranking, $y_w$ is the completion the labeler preferred to $y_l$, and $\sigma$ is the logistic function. This has the same form as the Bradley-Terry model used in the DPO paper: the difference of two scalar rewards is the log-odds that a labeler prefers one completion. The loss does not change if you add a constant to every reward. Before reinforcement learning they shift the reward so that labeler demonstrations score 0 on average. They train a 6 billion parameter reward model even when the policy is 175 billion. A 175 billion reward model was unstable, and they needed it as a value function during reinforcement learning. The reward-model set is about 33,000 prompts.

3. PPO. The reward model's scalar is the reward. Proximal policy optimisation (Schulman et al., 2017) fine-tunes the supervised policy to increase that reward. The PPO prompt set is about 31,000 prompts, from the API only, with no fresh human label on those prompts. The human signal is entirely inside the reward model. Steps 2 and 3 can be repeated. In their run, most comparison data came from the supervised policies, and some from the PPO policies.

On their test prompts, from customers held out of training, labelers preferred 175B InstructGPT to 175B GPT-3 $85\pm 3\%$ of the time, and to few-shot 175B GPT-3 $71\pm 4\%$ of the time. Held-out labelers, who had produced none of the training data, preferred InstructGPT to GPT-3 at about the same rate in a preliminary check. The paper says more work is required for broader groups of users, and for inputs where people disagree. That sentence is the limit on reading "human feedback" as "people".

The alignment tax is reduced, not defined away, by PPO-ptx: mix the PPO update with a term that raises the log likelihood of the pretraining distribution. They report that this mix cut the regressions on those public sets without giving up the labeler preference scores.

What the objective is, in one line: make the policy produce completions that the reward model scores highly, on the API-like prompts in the PPO set, where the reward model was fitted to these labelers' rankings, starting from a policy that imitates their demonstrations. It is not a direct factual-correctness loss; truthfulness enters through the labelling instructions and preferences. Outcomes are a separate question: the paper reports about twice as many true-and-informative TruthfulQA answers as GPT-3, closed-domain hallucination rates of 21% versus 41%, and about 25% fewer toxic outputs when prompted to be respectful. Those measured gains do not make the loss an independent fact-checker.

The objective DPO writes in closed form

Rafailov, Sharma, Mitchell, Ermon, Manning and Finn start from the same third-stage objective that RLHF uses and that DPO's section 3 writes as

$\beta$ sets how far the policy may move from the reference, which they take to be the SFT model. The KL term is there for two reasons they name: the reward model is only reliable near the distribution it was trained on, and without the penalty the policy collapses onto a few answers that score high. Because text is discrete, that objective is not differentiated directly. The usual move is to fold a sampled KL term into the scalar reward used for PPO,

and maximise it with PPO. That is the pipeline they call complex and often unstable: a reward model, samples from the policy, and a reinforcement-learning loop.

For the original reward function r (not the already KL-adjusted PPO scalar), the optimal policy of the KL-constrained problem has the closed form:

$Z(x)$ sums over every possible completion, which is why this expression is not something you compute. Rearrangement gives the reward in terms of the policy:

Bradley-Terry only needs the difference of two rewards. $Z(x)$ depends on the prompt, not on which completion you are scoring, so it cancels. The probability that $y_1$ is preferred to $y_2$ becomes a function of the optimal policy and the reference policy alone. Maximum likelihood on a dataset of $(x,y_w,y_l)$ is then

There is no reward head and no PPO step. The policy is the reward model, up to the reparameterisation $r(x,y)=\beta \log ({\pi }_{\theta }/{\pi }_{\mathrm{r}\mathrm{e}\mathrm{f}})$. Two reward functions that differ by a function of $x$ only induce the same preferences under Bradley-Terry; that is their Lemma 1. The class of reward functions is not shrunk by hiding the reward inside the policy.

The gradient shows what a step does. It raises $\log \pi (y_w\mid x)$ and lowers $\log \pi (y_l\mid x)$, and it multiplies that push by $\sigma (\widehat{r}(x,y_l)-\widehat{r}(x,y_w))$. Pairs the implicit reward already orders correctly get a small weight. Pairs it currently gets wrong get a large one. They report that dropping this weight and using a bare probability ratio degenerates the model. $\beta$ is inside that weight: it is both the KL budget and the scale of the implicit reward.

The pipeline they give is short. Sample two completions from the reference for each prompt, have a person pick a winner, then minimise the loss. If you already have a preference set sampled from an SFT model, use that model as ${\pi }_{\mathrm{r}\mathrm{e}\mathrm{f}}$. If you do not, they initialise the reference by maximum likelihood on the preferred completions only, to reduce the shift between the unavailable true reference and the one in the loss. Their experiments, on sentiment, summarisation and dialogue, with models up to 6 billion parameters, find DPO at least as effective as PPO-based RLHF on those tasks. That is their comparison, not a result at 175 billion, which they did not run.

What to choose and why

Do not fine-tune with either method until the preference pairs are pairs a person actually ordered, from the prompt distribution you will serve. InstructGPT's PPO prompts were API prompts. A preference set of synthetic arguments about a different product is a different objective, and the KL term will not correct the mismatch. It only keeps you near the reference.

Keep the reference checkpoint. DPO's loss is a ratio against it. RLHF's KL term is a ratio against it. Delete the reference and you cannot reproduce the update, and you cannot tell whether a later policy has drifted.

Risks and limits

• The labeler pool is the objective. About 40 screened contractors, a prompt set that is over 96% English, and a preliminary held-out-labeler check. The paper asks for broader users and for cases where people disagree. Those cases are not solved by a larger $\beta$.
• The reward model can be gamed by the policy if the policy leaves the distribution the reward model saw. The KL term is the paper's control for that, not a proof that reward hacking cannot happen.
• SFT validation loss is the wrong early-stopping signal in their run. It rose after one epoch while preference ratings were still improving at 16 epochs.
• Ranking $K$ completions is not $K$ independent labels. Treating every pair as its own row overfit their reward model. The batching they describe is part of the method.
• DPO's published comparisons stop at 6B parameters and at sentiment, summarisation and dialogue. "At least as effective as PPO" is that claim.
• A missing weight in the DPO gradient is not a simplification. They found the unweighted ratio degenerates.
• No training run was executed for this article. Both papers were opened on 10 November 2026.
• No NetLeverage product behaviour is claimed.

Sources

1. Ouyang et al., Training language models to follow instructions with human feedback, arXiv:2203.02155 — https://arxiv.org/html/2203.02155 (opened 10 November 2026).
2. Rafailov et al., Direct Preference Optimization: Your Language Model is Secretly a Reward Model, arXiv:2305.18290 — https://arxiv.org/html/2305.18290 (opened 10 November 2026).

This article was originally published on 10 November 2026.