LoRA and QLoRA fine-tuning: mechanics, and when not to fine-tune

About this article: AI-assisted writing, reviewed against the cited sources. This is source research, not a lab validation of the procedures. Check the stated versions and your own recovery requirements before making changes.

LoRA does not train a second copy of the model. It freezes the weight matrix $W_0$ and trains a low-rank change $BA$, added to the same input. QLoRA stores that frozen matrix in 4-bit NormalFloat, dequantises it to 16-bit for the multiply, and still trains only the low-rank matrices. Both are ways to adapt a model you already have. Neither replaces retrieval when an answer must cite a current document, and neither is free of the base weights at inference.

An AI writing system produced this article. No adapter was trained. Every memory figure and score below is the one printed in the paper it is taken from. No NetLeverage training product is described.

Why it matters

A full fine-tune writes a new file the size of the original model, and the optimiser states for every parameter. Hu and colleagues use GPT-3 175B as the example: an independent fine-tuned copy per task does not fit an operations plan. Their LoRA run cut trainable parameters by about 10,000 times against full Adam fine-tuning of that model, and cut the GPU memory they report from 1.2 TB to 350 GB. QLoRA later puts a 65B-parameter fine-tune under 48 GB, against more than 780 GB for the 16-bit full fine-tune they compare with.

Those savings decide whether the job can run. They do not decide whether the job should. The update is a low-rank patch on weights that already exist. If the failure is a missing source document, a wrong retrieval, or a test that does not match the training questions, fine-tuning is not automatically the appropriate correction; training on a document can expose a model to it but does not provide a reliable current-source lookup.

How it works

The update LoRA is allowed to learn

Hu, Shen, Wallis, Allen-Zhu, Li, Wang, Wang and Chen start from a hypothesis, which they mark as a hypothesis: the change in weights during adaptation has a low intrinsic rank, in the same family of results as Aghajanyan, Gupta and Zettlemoyer on the intrinsic dimension of pre-trained models. For a frozen matrix $W_0$ of shape $d\times k$,

with $B$ of shape $d\times r$, $A$ of shape $r\times k$, and $r$ much smaller than both $d$ and $k$. $W_0$ gets no gradient. $A$ and $B$ do. The same input is multiplied by both, and the outputs are added.

$A$ is initialised from a random Gaussian. $B$ is initialised at zero. So $BA$ is zero at step 0 and the model begins as the pre-trained model. The product is then scaled by $\alpha /r$. They treat $\alpha$ as a constant in $r$: with Adam, tuning $\alpha$ is roughly tuning the learning rate if the initialisation is scaled to match, so they set $\alpha$ to the first $r$ they try and leave it. The point of the scale, citing Yang and Hu, is that you should not have to retune everything when $r$ changes.

At deploy time they form $W=W_0+BA$ and run the ordinary matmul. Both matrices have shape $d\times k$, so the merged layer has the same latency as a fully fine-tuned layer, by construction. Switching tasks is subtract $BA$, add $B^{\prime }{A}^{\prime }$. That only works if you still have $A$, $B$ and $W_0$. A merged file with the originals thrown away is just a full checkpoint again, and you cannot peel the task back off.

They apply this, in most of their experiments, only to the query and value projections $W_q$ and $W_v$, and they freeze the MLP. The count of trainable parameters is $2\times {\widehat{L}}_{\text{LoRA}}\times d_{\text{model}}\times r$, where ${\widehat{L}}_{\text{LoRA}}$ is how many weight matrices are adapted. Attention projections are treated as $d_{\text{model}}\times d_{\text{model}}$ even though the implementation slices them into heads. On GPT-3 175B they say a rank of one or two was enough for the tasks they ran, against a full dimension of 12,288. That is a result on those tasks, not a default for every later model.

The memory win is the optimiser. Adam keeps state per trainable parameter. Frozen weights need no gradient and no optimiser state. They put the saving at up to two-thirds of VRAM when $r$ is far below $d_{\text{model}}$. The storage win is the checkpoint: with $r=4$ and only query and value adapted, they report the adapted checkpoint shrinking from about 350 GB to 35 MB, roughly 10,000 times. The footnote is the part operators miss. You still deploy the 350 GB base. One hundred tasks are $350\text{ GB}+100\times 35\text{ MB}$, about 354 GB, against $100\times 350\text{ GB}$. The base is not optional.

On the same GPT-3 175B setup they report training throughput moving from 32.5 tokens per second per V100, for full fine-tuning, to 43.1 with LoRA, the same number of weight shards. They call that a 25% speedup (the printed rates are 32.6% more tokens per second, equivalent to about 24.6% less time per token), from not forming gradients for most parameters. Those rates are their measurement on V100s.

If you set $r$ to the rank of the original matrices and you adapt every weight, including biases, they say you roughly recover full fine-tuning. Rank is then no longer a saving. It is the knob that spends parameters when the task is hard enough that a rank-4 patch is the wrong model of the change. They also record a serving limit: once $BA$ has been absorbed into $W$, a single forward pass cannot easily mix examples that need different adapters. Leaving the adapters unmerged, and picking one per example, works when the extra latency is acceptable.

Their GPT-3 175B table is the quality claim, and it is task-specific. On WikiSQL LoRA matched full fine-tuning within the quoted fluctuation; on MNLI-matched and SAMSum it scored higher, with BitFit and a larger adapter close on MNLI. The reported fluctuations were about $\pm 0.5$ points on WikiSQL, $\pm 0.1$ on MNLI-matched, and about $\pm 0.2/0.2/0.1$ on the three SAMSum ROUGE scores. That is one paper, those sets, that model. It is not a guarantee that a rank-8 adapter on a 2026 checkpoint will beat a full fine-tune of it.

What QLoRA freezes, and in how many bits

Dettmers, Pagnoni, Holtzman and Zettlemoyer keep the LoRA structure and change the storage of $W$. The base model is quantised to 4-bit and stays frozen. Gradients flow through it into the adapters, which stay in higher precision. They report fine-tuning a 65B model on one 48 GB GPU. Their limitations section explicitly says they did not establish a match to full 16-bit fine-tuning at 33B and 65B. The controlled precision comparisons are on smaller models, including LLaMA 7B MMLU; the largest-model memory result must not be presented as a controlled no-quality-loss result. The memory line they print is from more than 780 GB down to under 48 GB.

Three mechanisms do that.

4-bit NormalFloat. Ordinary min-max quantisation wastes bins when one outlier sets the scale. They chunk the tensor into blocks and quantise each block with its own constant. NormalFloat goes further: bins are quantiles of a standard normal, so each bin expects the same count if the weights are normal and zero-centred. A symmetric grid has no exact zero, and padding wants an exact zero. NF4 is built asymmetric — $2^{k-1}$ negative levels and $2^{k-1}+1$ positive ones, then one of the two zeros is dropped — so zero is representable and all $2^k$ codes are used. They call this information-theoretically optimal for zero-centred normal data. It is not a claim about weights that are not distributed that way.

Double quantisation. A block size of 64 with 32-bit constants costs $32/64=0.5$ extra bits per parameter. They quantise those constants again: FP8, block size 256, after subtracting the mean so the constants sit around zero. The overhead becomes $8/64+32/(64\times 256)=0.127$ bits per parameter, a cut of 0.373 bits. They put that at about 3 GB on a 65B model. The weight block size stays 64. The constant block size is 256. They say they saw no damage from the 8-bit second stage, in line with earlier quantisation results they cite, not as a universal measurement.

Paged optimisers. Long sequences spike past the GPU during gradient checkpointing. They allocate optimiser state with NVIDIA unified memory, so a spike pages that state to CPU RAM and pages it back for the update. It is the same idea as paging to disk, pointed at CPU memory. It does not shrink the steady-state footprint. It is designed to let eligible optimiser-state pages leave GPU memory under pressure. The paper says it does not provide hard paging measurements, so this is not a guarantee that every memory spike succeeds.

The forward equation they write for one layer is a dequantised base matmul plus a LoRA matmul, both in BF16:

Storage is NF4. Computation is BF16. Only the adapter weights receive a gradient. The backward pass still dequantises $W$ to BF16 to form the activation gradient that reaches the adapter. "The base is frozen" does not mean the base is absent from the backward pass.

They also correct a memory myth with one 7B LLaMA run on FLAN v2, batch size 1, adapters at about 0.2% of the original weights. The LoRA parameters were 26 MB. The LoRA input gradients were 567 MB. Gradient checkpointing brought those gradients to an average of 18 MB per sequence. The 4-bit base was 5,048 MB. The adapter file is the small object. The activation gradient and the quantised base dominate the GPU.

Their chatbot numbers need the judge attached. Guanaco 65B, 41 GB at inference in their table, is scored by GPT-4 on the Vicuna prompts, Elo $1022\pm 1$ over 10,000 random orderings. They describe the best Guanaco as 99.3% of ChatGPT on that benchmark after 24 hours on one GPU, and a 33B-class run as 97.8% in under 12 hours on one consumer GPU. GPT-4 is deciding the matches. The paper also ran a Mechanical Turk human study. Its human Elo scores were 1176 for GPT-4, 1023 for Guanaco-65B, 1009 for Guanaco-33B and 916 for ChatGPT-3.5 Turbo. Agreement with GPT-4 was moderate at the system level (Kendall tau 0.43) and weaker per example (Fleiss kappa 0.25); neither judge can be substituted silently for the other. They also report that a 9,000-example OASST1 set beat a 450,000-example subsample of FLAN v2 on chatbot performance, and that a strong MMLU score did not imply a strong Vicuna score, or the reverse. Dataset match beat dataset size in that comparison.

What to choose and why

Keep the base revision, the adapter, $\alpha$, $r$, and which matrices were adapted, as one bundle. A 35 MB file from the LoRA footnote is meaningless against a different 350 GB checkpoint. QLoRA adds the quantisation: NF4, block size 64, double-quant block size 256. Restore a different quantisation and you are not running the model that was trained.

Do not fine-tune to paper over a prompt or a retrieval bug you have not measured. The method's own scope is a low-rank change to frozen weights, evaluated on the tasks in these two papers. If a held-out set from the real task does not move, more rank will not invent the missing evidence. If it moves on the training questions and not on the questions users ask, QLoRA's MMLU-versus-Vicuna split is the pattern you are in.

Risks and limits

• Low rank is a hypothesis about the update, confirmed on the tasks they published. Rank 1 or 2 on GPT-3 175B is their report, against dimension 12,288, not a default.
• The base stays in the deploy footprint. 35 MB is the adapter. 350 GB, in their footnote, is still required. QLoRA's 26 MB adapter sat next to a 5,048 MB 4-bit 7B base.
• Frozen 4-bit weights are still dequantised in the backward pass. Only $\partial E/\partial L_i$ is stored as a weight gradient. The matmul that produces it uses BF16.
• NF4 assumes a normal, zero-centred weight. The paper's optimality claim is for that distribution. Outliers are why they quantise in blocks of 64, not why every tensor is normal.
• Paged optimisers move spikes to CPU memory. They do not reduce the size of a spike that does not fit in CPU RAM either, and the paper does not claim they do.
• Guanaco's comparison with ChatGPT depends on judge and prompts. Report the GPT-4 and human evaluations together with their limited agreement.
• No adapter was trained for this article. Figures are from the two papers as opened on 10 November 2026.
• No NetLeverage product behaviour is claimed.

Sources

1. Hu et al., LoRA: Low-Rank Adaptation of Large Language Models, arXiv:2106.09685 — https://arxiv.org/html/2106.09685 (opened 10 November 2026).
2. Dettmers, Pagnoni, Holtzman and Zettlemoyer, QLoRA: Efficient Finetuning of Quantized LLMs, arXiv:2305.14314 — https://arxiv.org/html/2305.14314 (opened 10 November 2026).

This article was originally published on 10 November 2026.