Quantisation: GGUF, GPTQ and AWQ, and what the quality numbers measure

About this article: AI-assisted writing, reviewed against the cited sources. This is source research, not a lab validation of the procedures. Check the stated versions and your own recovery requirements before making changes.

A "4-bit model" is three decisions stacked on top of each other. Something chooses an integer for each weight. Something records those integers, plus the scales that turn them back into numbers. Something multiplies the restored weights by activations that were never stored at 4 bits. GPTQ and AWQ are procedures for the first decision. GGUF is a file format for the second. The third is why a smaller file can answer faster on a memory-bound decode, and why weight-only quantisation does not by itself change the dtype of a long conversation's key-value cache.

An AI writing system helped prepare this article for NetLeverage. Nothing below was executed in a lab. Perplexity cells, latencies and file sizes are the sources' own, on the models they name. This article makes no NetLeverage product claim. The measurement used here is perplexity, or a latency table, and it is only as wide as that table.

Why it matters

Weights are the part of a large model that does not change during a conversation. For OPT-175B in 16-bit, Frantar et al. needed five 80 GB A100 GPUs to hold the model. Their 3-bit GPTQ copy, with embeddings and the output layer left in FP16, was about 63 GB, plus about 9 GB of key-value history at a 2,048-token maximum, and it fit on one A100. That is the GPTQ paper's reported deployment result for that model, first published in 2022. It is also a reminder that quantising the weights does not quantise the cache. A long context still allocates the cache described in the key-value article, at whatever width the runtime keeps for keys and values.

The quality side is easy to over-read. On WikiText-2, the same paper's 4-bit GPTQ OPT-175B scores 8.37 perplexity against 8.34 for the 16-bit model. Round-to-nearest on the same grid scores 10.54. Three bits of round-to-nearest scores 7,300. The method, the bit-width and the grouping are doing different jobs. A filename that only says "4-bit" does not say which job was done.

How it works

What is being stored

Post-training quantisation, in both papers, starts from a finished model and a small calibration set. It does not retrain the network. The weights of a linear layer are replaced by integers on a grid, plus a scale (and, in several schemes, a minimum) so that a later kernel can rebuild an approximate floating-point value. Activations stay in a wider format. Lin et al. call this setting W4A16 when the weights are 4-bit and the activations are 16-bit. The multiply itself is a floating-point multiply after a dequantisation that they fuse into the kernel, so the 4-bit values are not written back to memory as 16-bit weights on every step.

Decode is the step that benefits. Lin et al. profile Llama-2-7B on an RTX 4090 and report that token generation is memory-bound, that weight bytes dominate activation bytes, and that moving the weights to 4-bit raises arithmetic intensity by about four times. Frantar et al. make the same bandwidth argument for OPT-175B and publish a kernel that dequantises inside the matrix-vector product. Their Table 6 is batch size 1, sequences of length 128: on an A100 the 3-bit model takes 71 ms per token against 230 ms for FP16, and the FP16 run used five GPUs while the 3-bit run used one. On A6000 GPUs the pair is 130 ms on two cards against 589 ms for FP16 on eight. The speedup is real in that table. It is a comparison with a multi-GPU 16-bit baseline, and it is for their kernel, not for every 3-bit file.

Round-to-nearest, and why the large models break it

The shared baseline is round-to-nearest: each weight goes to the closest grid point, with a scale chosen per row or per group. At 4 bits on OPT-175B it costs 2.2 WikiText-2 perplexity (8.34 to 10.54). At 3 bits it collapses, 8.34 to 7,300, and LAMBADA accuracy in their table falls to 0. BLOOM-176B is less fragile at 4 bits (8.11 to 8.37 for round-to-nearest) and still collapses at 3 bits (571 on WikiText-2). A uniform grid with no error compensation is a different algorithm from "4-bit" as a slogan.

GPTQ: compensate with the layer's curvature

GPTQ quantises one layer at a time. The layer's job, following Optimal Brain Quantization, is to keep the layer output close when only the weights change. For one row of weights the squared error is quadratic, with Hessian $2X{X}^{\top }$ built from the layer inputs on the calibration set. Quantising a single weight to the nearest grid point has an optimal correction for the weights that remain. Doing that greedily, one weight at a time, is accurate and far too slow at 175 billion parameters. GPTQ quantises a block of columns together, keeps the inverse-Hessian information in a Cholesky factor, and applies the correction at the end of the block. The published procedure uses the inputs that actually come out of the layers already quantised, not the inputs of the untouched 16-bit model. They damp the Hessian with $\lambda I$. Their GPTQ experiment uses 128 randomly selected 2,048-token segments from C4 for calibration. OPT-175B and BLOOM-176B each took about four GPU-hours.

The WikiText-2 cells for OPT, 16-bit then 4-bit GPTQ then 3-bit GPTQ, are 8.34, 8.37 and 8.68. Across WikiText-2, PTB and C4, their 4-bit GPTQ models stay within 0.25 perplexity of the 16-bit baseline (the PTB gap for OPT-175B is 12.01 to 12.26, which is the 0.25). At 3 bits the gap is larger: OPT-175B moves from 8.34 to 8.68 on WikiText-2, 12.01 to 12.68 on PTB, and 10.13 to 10.67 on C4. Grouping tightens it. A group of 128 weights, which they cost at about 0.15 extra bits, brings OPT-175B at 3 bits to 8.45, 12.37 and 10.36 on those three sets. LAMBADA for 4-bit GPTQ OPT-175B is 76.80 against 75.59 for the baseline, so perplexity and LAMBADA do not always move together: 4-bit GPTQ is slightly worse on perplexity here and slightly better on LAMBADA. They also report that, in this family, larger models were generally easier to quantise than smaller ones, with OPT-66B as an exception they tie to dead units in early layers. That trend is about OPT and BLOOM in 2022. It is not a promise about a later architecture.

AWQ: protect the channels that the activations use

Lin et al. start from a different observation. On OPT-6.7B at 3 bits with groups of 128, keeping about 1% of weight channels in FP16 drops perplexity from 43.2 to 13.0 when those channels are chosen by activation magnitude. Choosing the same fraction by weight magnitude does not. Mixed precision is awkward for a kernel that wants one packing. They show that scaling a salient channel by $s>1$ before the grid is chosen reduces that channel's relative rounding error, because the scale $\Delta$ of the group often stays put while the weight grows. For the same OPT-6.7B, $s=1$ (plain round-to-nearest) scores 23.54 and $s=2$ scores 11.92. Larger $s$ starts to hurt the other channels, because $\Delta$ does move. The method they ship searches one exponent $\alpha$ in $[0,1]$ so the per-channel scale is a power of the average activation magnitude, on a grid of 20 values, using a small calibration slice of the Pile, plus a clip that reduces mean squared error. The inverse scale can be folded into the previous operator, so the network's function, before quantisation, is unchanged by the scaling.

Their Table 4 is WikiText-2 perplexity. Llama-2-7B at 4 bits, group 128: FP16 5.47, round-to-nearest 5.73, GPTQ 5.69, GPTQ with the reorder trick 5.63, AWQ 5.60. Llama-2-70B at the same setting: 3.32, 3.46, 3.42, 3.43, 3.41. At 3 bits the spread is wider. Llama-2-7B: FP16 5.47, round-to-nearest 6.66, GPTQ 6.43, AWQ 6.24. LLaMA-7B at 3 bits is the cell where GPTQ without reorder is weak (8.81) and reorder restores 6.53, with AWQ at 6.35. They argue that GPTQ's reconstruction can overfit the calibration set. That is their interpretation of their own comparisons, including instruction-tuned and multimodal models in the same paper. It is not an independent re-run. On Mixtral-8x7B their AWQ numbers are 5.94 in FP16, 6.05 at 4-bit group 128, and 6.52 at 3-bit group 128. TinyChat, their runtime, is what turns the 4-bit weights into the 3.2–3.3× average speedup they report against a Hugging Face FP16 baseline on desktop, laptop and mobile GPUs. The perplexity table and the speedup are different measurements.

GGUF: a container, then a quantiser that writes into it

The GGUF specification from the ggml project describes a file, not a rounding algorithm. The header begins with the four bytes G, G, U, F, then a version integer (version 3 is the structural version that the spec text says introduces big-endian support), then a tensor count and a count of metadata key-value pairs. Metadata is typed, so a new field can be added without a new header layout. Each tensor has a name, a shape and a ggml_type code. The spec points readers at ggml.h for the live list of codes, and it says that list gains values over time. general.file_type records the majority tensor type. general.quantization_version is a separate integer, required once any tensor is quantised, and the spec's own example is that a scheme can still be named Q5_K while the version number moves. Alignment defaults to 32 when the field is absent. The design goals named in the spec are a single file, mmap loading, and enough metadata to load the model without an external config. GGUF replaced the earlier GGML, GGMF and GGJT layouts for that reason. Little-endian is the assumption when the file does not say otherwise. The spec notes that, at the time it was written, a big-endian file was not self-describing.

llama.cpp's quantize README, fetched from master on 10 November 2026, splits the work in two. Convert a checkpoint to a high-precision GGUF, then run llama-quantize to a named type such as Q4_K_M. The README warns that --allow-requantize can severely reduce quality relative to quantising from 16-bit or 32-bit. --pure disables the k-quant mixtures and writes every tensor as one type. An importance matrix is optional and is applied to tensors you name. Those flags are why two files that both say Q4_K_M can differ: one left the output tensor in a wider type, one requantised an already quantised file, one used an importance matrix.

The type code and the file average are different bit-widths. The project wiki page "Tensor Encoding Schemes", dated 31 July 2024, describes Q4_0 as blocks of 32 weights with a scale, $w=q\times \text{scale}$, and Q4_K as a super-block whose scales and minima are themselves quantised. The page says it is a starting point rather than a definitive writeup, and one line is marked "to be confirmed", so this article uses it for the block idea and the type names, and uses the README for measured file averages. On meta-llama/Llama-3.1-8B, that README table lists F16 at 16.0005 bits per weight and 14.96 GiB, Q8_0 at 8.5008 bits per weight and 7.95 GiB, Q4_K_M at 4.8944 bits per weight and 4.58 GiB, and Q2_K at 3.1593 bits per weight and 2.95 GiB. The 4.8944 exceeds a flat four bits because storage includes block-scale overhead and a mixture of tensor types. Comments in quantize.cpp on the same unpinned master give a different kind of number, for a different model name: "+0.1754 ppl @ Llama-3-8B" beside Q4_K_M, "+0.0026" beside Q8_0, and "+3.5199" beside Q2_K. Those strings are maintainer annotations. They are not the Llama-3.1-8B table, and they do not name the evaluation set in the comment itself.

The same README table shows that generation and prompt-processing throughput can move in opposite directions. Text generation at 128 tokens is 29.17 tokens/s for F16, 71.93 for Q4_K_M, 50.93 for Q8_0 and 79.85 for Q2_K. Within those four selected entries, generation throughput rises as average bit-width falls. Prompt processing at 512 tokens instead falls from 923.49 tokens/s for F16 to 784.45 for Q2_K. Those four entries do not show a reversal in decode throughput as bits decrease; other encoding families in the full README table must be compared separately. The README section does not name the machine. Treat the column as evidence that bit-width, kernel and phase (prefill versus decode) move the speed in different directions, and remeasure on the machine you will use.

Weight quantisation also leaves the cache alone. Frantar et al. still budgeted about 9 GB of key-value state for 2,048 tokens of OPT-175B after the weights were 3-bit. A GGUF file of quantised weights does not, by the format description above, store that cache. A long conversation can be limited by the cache while the weight file looks small.

What to choose and why

Match the baseline before you quote a delta. GPTQ's 0.03 WikiText-2 gap is OPT-175B at 4 bits against that model's own 16-bit run. AWQ's 0.13 gap is Llama-2-7B at INT4-g128 against 5.47. The llama.cpp comment's +0.1754 is Llama-3-8B and does not say the dataset. These are three results.

Risks and limits

• 3-bit round-to-nearest is a documented failure on OPT-175B and BLOOM-176B in Frantar et al. A file named for 3 bits still needs the method and the group size.
• GPTQ and AWQ disagree, on purpose, about calibration. AWQ's authors say GPTQ can overfit the calibration set, and their Table 4 is the comparison they offer. A later runtime can change either method. Re-evaluate on the task you will serve.
• Perplexity is the sensitive metric both papers prefer, and it is not a task score. LAMBADA in the GPTQ table does not move in lockstep with WikiText-2. A chat rubric can move differently again.
• Speedups are kernel and topology specific. The 3.25× figure compares one A100 running 3-bit OPT-175B with five A100s running FP16. It does not say a 4-bit file is 3.25× faster than a 16-bit file on one GPU.
• The GGUF documents on master are unpinned. Type codes, the Q4_K_M mixture and the Llama-3.1-8B table can change after 10 November 2026. The wiki page is explicitly provisional and dated 31 July 2024.
• Requantising a quantised file is a second rounding. The llama.cpp README says the quality loss can be severe.
• The cache, the activations and any multimodal projector are separate budgets. The README's own guidance is to leave multimodal projectors in a wide type such as Q8_0 or BF16 because they prepare the inputs the language model sees.
• No lab run and no NetLeverage product behaviour are claimed.

Sources

1. Frantar, Ashkboos, Hoefler and Alistarh, GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers, arXiv:2210.17323 — https://arxiv.org/abs/2210.17323 (HTML via ar5iv, opened 10 November 2026). Layer-wise Hessian quantisation, OPT and BLOOM perplexity, memory and latency.
2. Lin et al., AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration, arXiv:2306.00978 — https://arxiv.org/abs/2306.00978 (HTML via ar5iv, opened 10 November 2026). Activation saliency, scaling, and the Llama-2 perplexity table.
3. ggml project, GGUF specification — https://github.com/ggml-org/ggml/blob/master/docs/gguf.md (raw file on master, opened 10 November 2026). Container layout, version, per-tensor types, quantisation version.
4. llama.cpp, tools/quantize/README.md — https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md (raw file on master, opened 10 November 2026). Convert-then-quantise, Q4_K_M as a mixture, Llama-3.1-8B size and speed table. quantize.cpp on the same branch supplies the maintainer perplexity comments, which this article treats as comments.
5. llama.cpp wiki, Tensor Encoding Schemes — https://github.com/ggml-org/llama.cpp/wiki/Tensor-Encoding-Schemes (page dated 31 July 2024, opened 10 November 2026). Block and super-block description. The page presents itself as a starting point.

This article was originally published on 10 November 2026.