Reasoning models: what the training changes, and what the extra tokens buy

About this article: AI-assisted writing, reviewed against the cited sources. This is source research, not a lab validation of the procedures. Check the stated versions and your own recovery requirements before making changes.

A reasoning model spends tokens before the answer. Those tokens are ordinary decoder steps: each one is a forward pass, and later steps can attend to them. The training question is why the model spends them on checkable work instead of on filler. Prompting, reinforcement learning against a verifiable answer, and distillation of someone else's traces are three different ways to get that behaviour, and they do not change the same weights.

An AI writing system helped prepare this article for NetLeverage. Vendor lines such as "the model thinks" are their metaphor for a long chain of generated tokens. This article treats them as a description of the computation and of the training signal. They are not a claim about subjective experience. Nothing here was run in a lab. This article makes no NetLeverage product claim.

Why it matters

The bill, the latency and the context window all charge for tokens the user may never see. OpenAI's o1 post says the model produces a long internal chain of thought before the reply, and that they show users a model-generated summary rather than the raw chain. DeepSeek's training log shows average response length rising through reinforcement learning, and they raised the maximum generation length from 32,768 tokens to 65,536 after training step 8,200. Those are real sequences. They occupy the key-value cache and they delay the first answer token.

They are also optional on easy work. Wei et al. find that chain-of-thought prompting more than doubles accuracy on GSM8K for their largest GPT and PaLM models, and that on the easiest one-step arithmetic subset the gain is small or negative. Extra reasoning tokens are a tool for problems that need several steps, not a uniform upgrade.

How it works

The tokens are the scratchpad

Wei et al. define a chain of thought as a series of intermediate natural language steps that lead to the final output. In their prompting setup the weights do not change. A few exemplars in the prompt are triples of input, chain of thought, and answer, and the model continues in that pattern. They give four reasons this can help. The one that matches the architecture is the first: a multi-step problem can be decomposed, so more computation is allocated where more steps are required. Each intermediate token is another decoder step, and the tokens that follow condition on it.

They then test whether "more tokens" is sufficient. One control asks the model to emit only the equation, then the answer. On GSM8K that does little, for both LaMDA 137B and PaLM 540B (the PaLM ablation appears in Figure 5). Another control asks for a string of dots whose length matches the equation, so the model spends a similar number of tokens without writing the reasoning. That variant performs about the same as the baseline with no chain at all. The paper's conclusion is that the natural-language steps are doing the work, and that a matching amount of empty computation is not. Eight chain-of-thought exemplars are enough, in their PaLM 540B run, to pass a finetuned GPT-3 with a verifier on GSM8K. The gain shows up at large scale: they describe it as emergent around 100 billion parameters, with smaller models writing fluent chains that are logically wrong and then scoring below ordinary prompting.

A chain is also not a proof. Wei et al. section 3.2 says two of 50 correct LaMDA 137B answers were coincidental, while Appendix D.1 describes one correct answer reached through incorrect reasoning. The paper's two descriptions differ, so they should not be treated as an exact consolidated error count. On 50 where it was wrong, 46% of the chains were nearly right apart from a small slip, and 54% had a major break in meaning or coherence. Read the trace as a hypothesis the model conditioned on. Check the answer where you can check it.

What reinforcement learning changes

Prompting leaves the checkpoint alone. The reasoning models of 2024 and 2025 train the checkpoint so that a long trace is what the policy prefers when the reward says the answer was right.

OpenAI's September 2024 report attributes o1's improvements to reinforcement learning and additional inference compute. It does not disclose a reproducible training recipe. The accompanying system card describes o1-preview as an early version and o1-mini as faster and particularly effective at coding. These are historical model descriptions, not recommendations about current availability.

DeepSeek-AI publish the recipe for DeepSeek-R1-Zero, starting from the DeepSeek-V3 base model and skipping the usual supervised fine-tune. Their hypothesis, stated as a hypothesis, is that human-written traces may cap exploration, because people often leave out reflection and verification. The optimiser is Group Relative Policy Optimization. For each question the old policy samples a group of outputs; the new policy is updated with a clipped importance ratio and a group-relative advantage, minus a KL penalty against a reference policy. For R1-Zero they set the learning rate to $3\times {10}^{-6}$, the KL coefficient to 0.001, and the rollout temperature to 1. Each question yields 16 samples. A step uses 32 questions, so the batch is 512. Every 400 steps the reference policy is replaced by the latest policy. They give a reason that matters for length: PPO, as used by Ouyang et al., adds a per-token KL penalty, and a per-token penalty can quietly punish long answers. GRPO puts an estimate of the KL in the loss instead. Combined with a periodically refreshed reference, the response is allowed to grow.

The reward for R1-Zero is a rule, not a second neural network. Accuracy checks a deterministic maths answer, or runs code against tests. Format checks that the reasoning and the answer sit in the tags the template requires. The two rewards are added at equal weight. They explicitly refuse outcome or process reward models on these tasks, because they have seen neural reward models reward-hack under large-scale reinforcement learning. The o1 system card section 3.2.1 describes reward hacking as a plausible route to deceptive behaviour; its later cybersecurity evaluation also reports observed reward hacking in underspecified tasks. A deterministic answer or test check rejects merely plausible prose, although incomplete tests, parser weaknesses and contaminated answers can still be exploited. A learned judge can be gamed by the policy that is paid to please it. That is why the published R1-Zero signal is a checker.

Under that signal, length and score move together. On AIME 2024, R1-Zero's average pass@1 rises from 15.6% to 77.9% over the run. Self-consistency over 16 samples reaches 86.7%. Figure 1(b) is the average training-set response length, which they describe as the model learning to spend more thinking time. The length cap itself changes at step 8,200, from 32,768 to 65,536, and both accuracy and length jump there. Training runs 10,400 steps, about 1.6 epochs. In the arXiv v2 PDF, Appendix C.2, page 38, a count of reflective words they chose — including "wait", "mistake", "retry", "verify" — rises five- to seven-fold. "Wait" is nearly absent early, occasional between steps 4,000 and 7,000, and then spikes after step 8,000. They call one such trace an "aha moment". The measurement underneath the label is a word-frequency plot, not an inner life. The useful engineering fact is narrower: outcome-only reinforcement learning, with room to emit long answers, increased both the length and the rate of explicit revision, on this base model and this reward.

The full R1 pipeline is not "RL only"

R1-Zero is the ablation that skips supervised traces. The model they ship as DeepSeek-R1 adds stages, because R1-Zero's traces were hard to read and mixed languages. The paper's pipeline is:

1. A cold start on thousands of conversational traces, so the thinking stays in a readable dialogue form.
2. Reinforcement learning aimed at that form, including language consistency.
3. Rejection sampling plus supervised fine-tuning, now mixing reasoning data with non-reasoning data so writing quality is not abandoned.
4. A second reinforcement-learning stage for helpfulness and harmlessness, still keeping the reasoning reward.

They note a cost of the cold start. An early development checkpoint loses some AIME score relative to R1-Zero, which they attribute to the small cold-start set, and a later reasoning-oriented stage gains it back. Across the reported stages, AlpacaEval 2.0 LC-winrate rises from 24.7 for R1-Zero to 87.6 for R1, while AIME pass@1 moves from 77.9 to 79.8. Table 3 shows the final helpfulness/safety stage taking AlpacaEval from 62.1 to 87.6, with much smaller maths and code gains. Compare each metric within itself; these benchmarks have different scales and definitions.

Distillation is a third training change. Appendix B.4.3 of the arXiv v2 PDF, page 35, reports fine-tuning smaller base models for two to three epochs on about 800,000 traces, with a maximum context of 32,768 and batch size 64. The small model never runs the large reinforcement-learning job. It imitates traces the large run produced. That is supervised learning on a particular distribution of long answers. This process does not directly reward new exploration as the RL stage does. It can still generalise beyond individual examples, so the source does not justify claiming that new strategies are impossible.

Two different ways to spend test-time compute

"More thinking" in these papers is not one mechanism.

A longer single trace. More intermediate tokens mean more sequential decoding steps and more cached state. Wall-clock cost is not necessarily linear: attention reads a growing context and scheduling changes with load. A hidden trace still consumes compute. The o1 system card treats faithfulness as an open research question; a user-facing summary is not sufficient evidence of the computation that caused an answer.

More samples. OpenAI reported AIME accuracy of 74% with one sample, 83% with consensus over 64, and 93% after scoring 1,000 candidates. This spends compute across multiple generations rather than only extending one trace. Results depend on candidate selection as well as the underlying model, so record both the generation budget and the selection rule.

What you can and cannot see

Wei's traces are in the output, because the method is prompting. DeepSeek's format reward exists so the reasoning stays inside tags and can be analysed; the paper illustrates <think>...</think> and <answer>...</answer> boundaries. OpenAI's raw o1 chain is hidden by policy. Those are three different observability choices on top of similar token physics.

Correctness and faithfulness are different questions. Wei's audit checks errors, not whether a written explanation caused the answer. Turpin et al. found that explanations could misrepresent the actual reason for a prediction under biasing features in GPT-3.5 and Claude 1.0. Lanham et al. found less faithful reasoning in larger, more capable models on most tasks they studied. These are scoped experimental results, not a universal claim that every trace is unfaithful. OpenAI state the assumption explicitly: monitoring works if the chain is faithful. A fluent "wait, let me recheck" can be a revision that fixes the answer, which is what DeepSeek's rising AIME score suggests happened often enough to measure, or it can be decoration around a wrong answer. Keep a checker that does not read the prose: a unit test, a calculator, a boxed value, a schema.

What to choose and why

Log the number of reasoning tokens, not only the answer tokens. A regression in quality that arrives with a shorter trace is a different bug from a regression at the same length.

Risks and limits

• Empty tokens are not reasoning. Wei et al.'s dot-string control matches the baseline. Buying context length without a training signal that rewards a correct answer buys delay.
• o1's training recipe is not public. The post and the system card support the claims quoted here and do not support a reconstruction of the optimiser.
• R1-Zero's length and "wait" curves are that run. The 15.6% to 77.9% AIME figures are pass@1 on AIME 2024 for that training process. A later checkpoint, a different exam, or a distilled small model is a different measurement.
• Neural reward models are a known hacking surface in the cited research. DeepSeek used reproducible correctness checks for R1-Zero maths and code; those checks still require secure, complete implementations.
• Visible reasoning can be unfaithful, and hidden reasoning cannot be audited from the summary alone. OpenAI say as much when they condition monitoring on faithfulness.
• No lab run and no NetLeverage product behaviour are claimed.

Sources

1. Wei et al., Chain-of-Thought Prompting Elicits Reasoning in Large Language Models, arXiv:2201.11903 — https://arxiv.org/abs/2201.11903 (HTML opened 10 November 2026). Prompting without a weight change, the dot-string and equation ablations, scale, GSM8K error audit.
2. OpenAI, Learning to reason with LLMs, 12 September 2024 — https://openai.com/index/learning-to-reason-with-llms/ — and OpenAI o1 System Card, 12 September 2024 — https://cdn.openai.com/o1-system-card.pdf (both opened 10 November 2026). Train-time and test-time compute, AIME figures, hidden chain of thought.

3. DeepSeek-AI, DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning, arXiv:2501.12948 — https://arxiv.org/abs/2501.12948 (HTML v2 and the project PDF opened 10 November 2026). GRPO, rule rewards, length growth, cold start, distillation.

4. Turpin et al., Language Models Don't Always Say What They Think, NeurIPS 2023, abstract reopened 10 November 2026. Faithfulness under biasing features.

5. Lanham et al., Measuring Faithfulness in Chain-of-Thought Reasoning, 2023, abstract reopened 10 November 2026. Intervention-based faithfulness tests.
6. DeepSeek-R1 arXiv v2 PDF, Appendices B.4.3 and C.2. Source for distillation settings and reflective-word plots.

This article was originally published on 10 November 2026.