GPU memory for inference: weights, the key-value cache, and how many requests fit

About this article: AI-assisted writing, reviewed against the cited sources. This is source research, not a lab validation of the procedures. Check the stated versions and your own recovery requirements before making changes.

Inference memory is two large piles and a small one. The weights stay for as long as the model is loaded. The key-value cache grows with every token of every live request and is what decides how many requests you can hold. Activations and CUDA graphs sit in what is left. A card with "80 GB" does not mean an 80 GB model. It means weights, plus a token budget, plus enough unused memory that the allocator does not die on the next graph capture.

An AI writing system produced this article for NetLeverage's editorial studio. The byte counts below are arithmetic from published model dimensions or figures copied from those papers. Nothing was allocated on a GPU for this article. No NetLeverage product is described.

Why it matters

A request that is refused, or a server that preempts a request and recomputes it, is usually a cache problem, not a compute problem. Kwon et al. show why. On a 13-billion-parameter model on a 40 GB A100, about 65 percent of memory was weights, close to 30 percent was the cache, and activations were the small remainder. For OPT-13B in FP16 they compute one token of cache as $2\times 5120\times 40\times 2$ bytes, which is 819,200 bytes, or 800 KiB (labelled 800 KB in the paper). At a 2048-token cap that is 1.6 GB for a single request. Their Table 1 leaves 12 GB for cache on that GPU, which they count as 15,700 slots. Divide 15,700 by 2048 and you can hold about seven requests if every one of them is at the cap, or many more if they are short. The card did not get smaller. The cache did.

Two levers change the arithmetic before you buy another GPU. Grouped-query attention stores fewer key and value heads than query heads, so the cache shrinks by that ratio. A lower cache dtype stores fewer bytes per element. Neither one changes the fact that concurrent long requests add.

How it works

Weights

A parameter stored in FP16 or BF16 occupies 2 bytes. OPT-13B in the vLLM paper is 26 GB of parameters on that basis (13 billion × 2 bytes is 26 billion bytes). As element-size arithmetic, FP8 is 1 byte per stored weight and a packed 4-bit integer is half a byte. These are not complete runtime footprints. Group scales and zero-points sit on top of that half-byte; the extra depends on the format's group size, which this article does not fix. Read the file, do not multiply "7B" by 0.5 and call it the whole footprint.

Tensor parallel cuts the weights per GPU and adds an all-reduce. Pipeline parallel cuts layers per GPU and can add latency. The vLLM tuning guide states both effects: tensor parallel leaves more room for cache on each GPU and costs synchronisation; pipeline parallel leaves more room and costs latency. The weights do not get smaller in total. They get spread out.

The cache, per token

For multi-head attention, each token stores a key and a value at every layer, each of width equal to the model dimension. In bytes, per token:

That is Kwon et al.'s OPT-13B sum: two tensors, dimension 5120, 40 layers, 2 bytes. The same token text at two positions is two entries. The vectors depend on the prefix.

Grouped-query attention, Ainslie et al., does not store a key and a value for every query head. Query heads are split into $G$ groups. Each group shares one key head and one value head. Multi-query attention is the case $G=1$. Multi-head attention is the case $G$ equal to the number of query heads. Going from multi-head to multi-query shrinks the cache, and the bytes loaded from it, by a factor of $H$, the query-head count. A model with $G$ groups shrinks that cache by $H/G$ relative to multi-head, because each head is $\text{dimension}/H$ wide and you keep $G$ of them. Eight groups, which they picked as a middle ground on T5-XXL, gives a reduction factor of $H/8$. That is eightfold only when $H=64$. Check the checkpoint. A model that was trained with grouped-query attention already has the small cache. You do not get the cut again by wishing.

Their reason is bandwidth. At decode, every new token reloads the weights and reloads every cached key and value. Fewer heads means fewer bytes of cache per step. They also note that for larger models the cache grows only with dimension, while parameter count grows with the square of dimension, so attention's share of the bandwidth problem shrinks as models get wider — and then standard sharding can waste memory by replicating a single multi-query head onto every partition. Grouped-query attention is their remedy for that waste. The quality claim in the paper is for uptrained T5 encoder-decoder models, not a blanket claim that every decoder-only checkpoint tolerates $G=8$.

Quantising the cache is a second factor on the bytes-per-element term. llama.cpp's server defaults --cache-type-k and --cache-type-v to f16 (2 bytes). The same flags accept bf16, f32, q8_0, q4_0, and others. A 4-bit cache is about a quarter of the FP16 cache for the stored elements, again before any scale metadata. The multi-GPU page forbids quantised cache types in experimental tensor-split mode: only f32, f16, and bf16 are accepted there, and a mix with quantised cache is a startup error.

Concurrency is a token budget

Add the per-token cost across every token that must stay resident.

If each of $N$ requests may reach length $L$, the pessimistic budget is $N\times L$ tokens. That is what you need if they can all be long together. It is also what older allocators reserved up front, which is why Kwon et al. found only 20.4 to 38.2 percent of cache memory holding real tokens.

A paged allocator charges for tokens that exist, plus at most one partial block per sequence. Their default block is 16 tokens, so the waste per live sequence is at most 15 positions, not a reservation out to the model's maximum length. Short requests then share the card. The limit is the sum of live lengths, not $N$ times the maximum, until a request actually grows.

That sum is still hostile. A night of long chats fills it. When it fills, vLLM's V1 guide says the default action is to preempt and recompute, not to swap. The documented warning names PreemptionMode.RECOMPUTE. The request still completes. The latency does not.

What the server actually reserves

vLLM's stable cache-config page, opened 10 November 2026, defines gpu_memory_utilization as the fraction of device memory for the model executor, default 0.92. The matching LLM docstring in the v0.23.0 API page says that fraction covers weights, activations, and the cache. Higher values enlarge the cache and the throughput, but too little headroom can cause out-of-memory errors. The V1 tuning page uses a shorter sentence: it "pre-allocates GPU cache using this percentage." Those two wordings are not identical. Use the field docstring of the version you run. Setting kv_cache_memory_bytes (the tuning page's --kv-cache-memory) overrides the fraction and ignores it. The tuning page warns that a pinned size is valid only for the same GPU and the same free memory at boot. Too small and you cap concurrency. Too large and allocation fails.

The limit is per vLLM instance. The cache-config page says a second vLLM instance on the same GPU is invisible to the first. Two processes at 0.92 will each believe they own 92 percent. The documented example is 0.5 per instance, adding up to one. Lower totals leave room for other users and overhead; successful colocation still requires an actual memory budget.

Free memory after weights is not all cache. The tuning page's startup path also estimates CUDA graphs. --enforce-eager skips that capture and slows steady-state decode. Activations during a wide prefill are larger than during a one-token decode. A budget that fits decode can still fail while the prompt is being scored. Kwon et al. call activations the small fraction in their serving figure. That figure is not a licence to set utilisation to 1.

llama.cpp sizes the cache from the context. --ctx-size defaults to 0, meaning the context stored with the model. --parallel is the number of slots, default automatic. The multi-GPU page, in its out-of-memory list, says a slot cache is allocated per concurrent sequence and that the cache is roughly proportional to context length. --kv-unified, the default when the slot count is automatic, uses one shared buffer. --kv-unified-per-slot N, if you set it and do not set a context size, sizes that pool to $N$ times the parallel count. Read that as a pool of tokens, not as $N$ guaranteed full-length replies, unless the pool is big enough for each of them.

A worked budget from the paper, then a grouped-query variant

Use only the published OPT-13B figures, then apply the grouped-query ratio as arithmetic.

If a checkpoint with the same layers and dimension used 8 query heads per key-value group, Ainslie et al.'s factor says the cache per token is one eighth, 100 KiB instead of 800 KiB. Scaling the paper's 15,700 slots by eight gives about 125,600 slots, or 61 full requests at a 2048-token cap. OPT-13B in the 2023 paper is not that checkpoint. The 100 KiB line is the formula applied to their dimensions, not a second measurement. A real grouped-query model also has its own layer count and head size. Read those from the config file and recompute. Do not reuse 800 KiB.

The same arithmetic is why a 4-bit weight store and a 16-bit cache is a common shape: the weights shrink by about four relative to FP16, and the cache does not. On a long context the cache becomes the larger pile. Shrinking only the weights does not fix a context-length problem.

What to choose and why

Log the number the server prints for cache blocks or token slots at startup, and compare it with your worst concurrent sum of prompt plus maximum new tokens. If the worst-case sum exceeds the slots, simultaneous long requests can exhaust the cache. It does not prove preemption will occur: actual live lengths, admission limits, and completion timing determine that.

Risks and limits

• The 65/30 split is one figure in one paper, a 13B model on a 40 GB A100. A small model on a large card is almost all cache. A large model on a tight card is almost all weights, and the cache holds a handful of tokens.
• gpu_memory_utilization defaults to 0.92 in the stable cache-config page opened that day. Later builds can change the default. The tuning page's wording and the field docstring do not describe the fraction in the same words.
• Pinned cache size skips measurement. The tuning page says an optimistic pin fails at allocation and a pessimistic pin caps throughput.
• Partial blocks still waste up to 15 tokens per sequence at block size 16. Thousands of tiny requests make that visible. It is not the old failure mode of reserving 2048 for each.
• Quantised cache and tensor split conflict in the llama.cpp multi-GPU document. The server will error rather than quietly upcast. Pipeline-style layer split does not carry that ban.
• Grouped-query quality is not free. Ainslie et al. uptrained T5 for about 5 percent of the original pre-training compute. Multi-query was less stable in their fine-tunes. Do not convert a production checkpoint by averaging heads unless you are prepared to uptrain and re-evaluate. The paper says they did not compare XXL grouped-query against a model trained from scratch.
• Activations during prefill are not the decode figure. A prompt of many thousands of tokens can peak above the steady decode budget. The vLLM guide describes chunked prefill as balancing compute-heavy prefill and memory-heavy decode; it does not establish an activation-peak reduction for every workload.
• No command was run. No NetLeverage capacity, card size, or service limit is stated.

Sources

1. Woosuk Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention, arXiv:2309.06180 — https://arxiv.org/html/2309.06180 (opened 10 November 2026). OPT-13B 800 KB/token, 26 GB weights, 12 GB cache, 15.7K slots, block size 16, waste figures.
2. Joshua Ainslie et al., GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints, arXiv:2305.13245 — https://arxiv.org/html/2305.13245 (opened 10 November 2026). Grouped-query definition, cache reduced by $H$ for multi-query and by the group ratio in general, 8 groups as their selected middle ground.
3. vLLM cache configuration — https://docs.vllm.ai/en/stable/api/vllm/config/cache/ (opened 10 November 2026). gpu_memory_utilization default 0.92, per-instance, and kv_cache_memory_bytes overriding it.
4. vLLM LLM API reference, v0.23.0 — https://docs.vllm.ai/en/v0.23.0/api/vllm/entrypoints/llm/ (opened 10 November 2026). Docstring: the fraction covers weights, activations, and cache.
5. vLLM Optimization and Tuning — https://docs.vllm.ai/en/stable/configuration/optimization/ (opened 10 November 2026). Preemption warning, recompute default, --kv-cache-memory caveat.
6. llama.cpp server README, https://github.com/ggml-org/llama.cpp/tree/master/tools/server, and multi-GPU documentation, https://github.com/ggml-org/llama.cpp/blob/master/docs/multi-gpu.md (opened 10 November 2026). Cache dtype default f16, context and slot flags, quantised-cache ban under tensor split.

This article was originally published on 10 November 2026.