KV cache and context windows: the memory cost of long conversations

About this article: AI-assisted writing, reviewed against the cited sources. This is source research, not a lab validation of the procedures. Check the stated versions and your own recovery requirements before making changes.

Once a token has been through a Transformer layer, the keys and values for that position can be kept. The next token reuses them instead of rebuilding the whole history. That store is the key-value cache, usually called the KV cache. Its size grows with every token still in the conversation, at every layer. A long chat is therefore a memory bill, separate from the size of the weights.

An AI writing system helped prepare this article for NetLeverage. Saying the model "remembers" the conversation means those stored keys and values are still available to attention. It is not a claim about subjective experience. Nothing below was run in a lab. Byte counts are the papers' own arithmetic or a multiplication of figures they print. This article makes no NetLeverage product claim.

Why it matters

Three different limits get called "context".

The trained window is the longest sequence the training run used. Llama 2 raised that length from 2,048 tokens in Llama 1 to 4,096, and the authors name longer chat histories as a reason. Past that length you are outside the setting they trained.

The live cache is the keys and values of the tokens actually in this request. It grows when the user sends another turn and when the model emits another token, and it can be freed when the request ends.

The allocator's reservation can be larger than the live cache. Kwon et al. describe systems that pre-allocate a contiguous chunk for the maximum length, for example 2,048 tokens, at the start of the request. Empty slots still occupy GPU memory.

Pope et al. separate the time spent moving tensors from high-bandwidth memory into the compute cores ("memory time") from the time spent on matrix multiplies ("compute time"). At small batches and short sequences, loading the weights dominates. At long sequences and large batches — their example is 2,048 tokens and beyond, with batch size 512 and beyond — loading the KV cache dominates. For a 500B-parameter-class model with ordinary multi-head attention, they put the cache at batch 512 and context 2,048 at 3 TB, three times the parameter memory. The chip then loads that cache from off-chip memory for every new token, while the arithmetic units wait.

Kwon et al. make the same point on a model that fits a single GPU. For OPT-13B in FP16 they compute one token of cache as 2 × 5,120 × 40 × 2 = 800 KB, and a full 2,048-token request as 1.6 GB. GPU cards they discuss top out at 80 GB. Even if every byte of that were cache, only a few tens of full-length requests would fit, and the weights still need room.

How it works

What gets stored

Scaled dot-product attention mixes a query against keys and then against values. In an autoregressive decoder the query at the newest position may attend to every earlier position, and to itself, and to nothing later. Shazeer spells out the serving consequence: during training the whole target is known, so those positions run in parallel; during generation, the token just sampled becomes part of the next input, so the steps are sequential.

A standard decode implementation therefore keeps, for each layer, the key vector and the value vector of every position already produced. Llama 2's appendix states that practice directly: cache the K and V pairs of previous tokens to speed up attention. The new step computes a query, a key and a value for the new token only, appends the new key and value to the cache, and runs attention against the whole cache.

Pope et al. name the two phases:

• Prefill. All input tokens are present, so one forward pass covers the whole prompt. The cache is filled for those positions.
• Decode. Each new token is one forward pass. The cache is read and then extended by one position.

Attention's extra floating-point work is often small beside the big matrix multiplies. Pope et al., citing Kaplan et al., note that an N-parameter decoder does about 2N matmul FLOPs per token (counting one multiplication and one addition as two FLOPs), and that attention's own FLOPs are a much smaller addition for a large model. The cache is different: unlike the weights, it is unique to each sequence. A batch of long chats multiplies memory capacity and memory traffic by the number of sequences, even when the FLOP count per token barely changes.

The byte formula, from a printed example

Kwon et al. write the OPT-13B cost as:

The factors are: one key and one value, a hidden width of 5,120, 40 layers, and 2 bytes for an FP16 number. That product is 819,200 bytes. They report it as 800 KB per token. Multiply by the 2,048-token maximum they use and the request is about 1.68 billion bytes, which they report as 1.6 GB.

Read the factors as a template:

A chat that has reached 1,280 tokens on that model holds $1280\times 819200=1048576000$ bytes of cache for that one request. That figure is this article's multiplication of the paper's per-token product, specific to OPT-13B in FP16 with multi-head attention. Recompute it from the checkpoint you actually load.

Why multi-query and grouped-query exist

Shazeer's incremental analysis is about bandwidth, not only capacity. Each decode step reloads K and V. He finds the ratio of memory traffic to arithmetic looking like $n/d+1/b$, with $n$ the length, $d$ the model width and $b$ the batch. When the length is comparable to the width, or the batch is 1, that ratio sits near 1 and the step is limited by memory bandwidth. Raising the batch helps the $1/b$ term, if the card has room for the larger cache. The $n/d$ term is the reload of K and V.

Multi-query attention leaves the queries with $h$ heads and gives every query head the same single key head and value head. Shazeer shows that this divides the troublesome $n/d$ term by $h$. On a WMT 2014 English–German Transformer and on a billion-word language model, with parameter count matched by widening the feed-forward layers, the paper's abstract reports a large decode speed-up and only minor quality loss. Those are 2019 models with a few hundred million parameters. Treat the speed-up as a demonstrated mechanism, and re-measure quality on any modern checkpoint before you assume the "minor" still holds.

Grouped-query attention, from Ainslie et al., sits between the two. Query heads are split into $G$ groups. Each group shares one key head and one value head. One group is multi-query attention. $G$ equal to the number of query heads is ordinary multi-head attention. Extending the paper's factor-of-H statement for multi-query attention, our arithmetic gives an $H/G$ reduction in key/value storage relative to multi-head attention, for the same layer count, head width and dtype without replication or allocator overhead. Here H is the query-head count. Ainslie et al. also note that a pure multi-query cut becomes more severe as models grow more heads, and that naive sharding can replicate the single key-value head once per partition, which spends the saving. Their uptraining recipe mean-pools the old key and value heads and continues pretraining for a small fraction of the original steps; at 5% they report T5-XXL quality close to the multi-head original and speed close to multi-query, on the summarisation, translation and question-answering sets in that paper. Figure 6 in the paper uses inputs of 2,048 and outputs of 512: moving from 1 group to 8 adds a modest time cost, and further groups cost more.

Llama 2 reports that choice across its model family; the 34B variants were described in the paper but not released. Table 1 marks grouped-query attention off for the 7B and 13B models and on for the 34B and 70B, all at a 4,096-token context. The appendix compares multi-head, multi-query and grouped-query on a 30B model trained for 150B tokens. Grouped-query matched multi-head on most of their tasks and beat multi-query on average. They also host the largest models on eight A100 GPUs with tensor parallelism. Multi-query then has fewer key-value heads than GPUs, so the cache is either copied to every GPU — which makes its memory look like an 8-head grouped-query cache — or split across the batch, the layout Pope et al. describe, which needs a batch larger than the number of shards. Llama 2 chose grouped-query for the 34B and 70B on that combination of quality and serving shape. Their Figure 24, output length fixed at 128 tokens on 8 × 80 GiB A100s, shows the multi-head ablation running out of memory at batch 1,024 with a 256-token context and at batch 128 with a 2,048-token context, while the multi-query and grouped-query runs still fit. In that figure the multi-query keys and values were duplicated, so its cache size matched grouped-query. Read the plot as "fewer distinct key-value heads delay the out-of-memory point", and read the duplication note before you quote it as a pure multi-query saving.

A conversation is a growing sequence

Llama 2's reason for a 4,096-token window is operational: longer histories in chat, plus summarisation and long documents. An ablation that changed only the training length, both runs at 150B tokens, improved SCROLLS, whose average input they give as 3.5k tokens, and did not degrade SQuAD. Table 17 reports these 2k-to-4k comparisons: HellaSwag 75.1 to 74.8, NQ 25.5 to 25.5, TQA 53.7 to 52.2, GSM8K 4.9 to 6.5, and HumanEval 7.9 to 7.3. Those mixed differences should not be read as improvement on every task. The window is doing work on long inputs. It is also a hard stop. Elsewhere they report a consistency method holding for 20 or more turns and then stopping because the maximum context length is reached. Turns are not the unit. Tokens are.

Each turn does two things to the cache:

1. The new user text is prefilled and its keys and values are appended.
2. Each generated token appends one more position during decode.

Our service-design inference from those mechanisms is as follows. If the service drops the cache at the end of the HTTP request, the next turn prefills the entire history again. Pope et al. define prefill as the parallel pass over every input token present at the start, so that cost scales with the whole transcript, not with the new message alone. If the service keeps the cache, the next turn can prefill only the new tokens and attend into the stored prefix, and the memory stays reserved for as long as the session is alive. Kwon et al. describe the cache as growing and shrinking with the request, and as freed when generation finishes. Which of the two service designs you run is an operations choice. The bytes per live token are not.

Shared prefixes are the one case where two sequences need not each pay the full history. Kwon et al. map several logical blocks onto one physical copy of a common prompt, with a reference count, and copy a block when a sequence needs to write its own continuation. Beam search and "n best" decoding are the same idea inside one request. That sharing is real only while the prefix tokens and their positions are actually the same. A chat template that inserts a different timestamp, or a tool result, into the middle of the history breaks the share from that point on.

Reservation, and why the formula underestimates a naive server

The formula counts tokens that exist. A contiguous allocator often reserves the maximum length immediately, because the cache has to sit in one span and the server does not yet know how long the answer will be. Kwon et al. list three wastes: slots reserved for future tokens, internal fragmentation when the reserved maximum exceeds the true length, and external fragmentation when neighbouring requests reserved different sizes. In the systems they profiled, 20.4% to 38.2% of KV-cache memory held actual token state. The rest was waste of one of those three kinds.

Their later design splits the cache into fixed-size blocks that need not be contiguous, allocates blocks as tokens arrive, and reports 2–4× the throughput of FasterTransformer and Orca at similar latency on their workloads, with a larger gap on longer sequences. The block layout itself is a serving topic of its own. For sizing a conversation, the point to take from that paper is narrower: budget the live tokens with the formula, and then ask whether your allocator charges you the maximum window for every request from the first token. If it does, a 4,096-token window costs 4,096 tokens of cache per concurrent chat even when the user has typed a sentence.

What to choose and why

Stay inside the context length the checkpoint was trained for. Llama 2's published window for that family is 4,096 tokens. A longer allocation does not create a longer trained window. Measure the per-token cost from the config you serve: key-value heads, head width, layers, and bytes per element. Do not reuse 800 KB, which is OPT-13B in FP16, as a constant for other models.

Risks and limits

• The 3 TB figure is Pope et al.'s case for a 500B-class multi-head model at batch 512 and context 2,048. It explains the scaling. It is not the footprint of a 13B model on one GPU.
• 800 KB and 1.6 GB are OPT-13B, FP16, up to 2,048 tokens, as Kwon et al. compute them. Change the dtype and the last factor changes. This article does not measure FP8 or INT8 caches.
• Quality claims for fewer key-value heads are tied to the training run. Shazeer's "minor" degradation is on 2019 translation and language-model setups. Llama 2's grouped-query comparison is a 30B ablation at 150B tokens, not a blanket result for every 70B checkpoint. Ainslie et al. evaluated encoder-decoder T5 models and note that decoder-only models were outside that study.
• Figure 24 is not a pure multi-query memory win. Llama 2 duplicated those keys and values across the eight GPUs, so the cache matched the grouped-query run.
• The 2–4× throughput is Kwon et al.'s evaluation against the systems and workloads in that paper. Longer sequences widened the gap there. It is not a guaranteed factor on another stack.
• Position encoding can make a naive truncation wrong. This article does not validate a scheme that deletes the start of a cache and keeps the tail. If you drop history, recompute or use a method whose paper you have checked for that model.
• No lab run and no NetLeverage product behaviour are claimed.

Sources

1. Shazeer, Fast Transformer Decoding: One Write-Head is All You Need, arXiv:1911.02150 — https://arxiv.org/abs/1911.02150 (HTML via ar5iv, opened 10 November 2026). Incremental decode reloads K and V; multi-query attention shares one key and value head.
2. Pope et al., Efficiently Scaling Transformer Inference, arXiv:2211.05102 — https://arxiv.org/abs/2211.05102 (opened 10 November 2026). Prefill versus decode, memory time versus compute time, the 3 TB cache example, multi-query partitioning.
3. Ainslie et al., GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints, arXiv:2305.13245 — https://arxiv.org/abs/2305.13245 (HTML via ar5iv, opened 10 November 2026). Grouped-query attention, cache scaling with key-value heads, uptraining results.
4. Touvron et al., Llama 2: Open Foundation and Fine-Tuned Chat Models, arXiv:2307.09288 — https://arxiv.org/abs/2307.09288 (HTML via ar5iv, opened 10 November 2026). Context 4,096, which sizes use grouped-query attention, and the eight-GPU memory notes.
5. Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention, arXiv:2309.06180 — https://arxiv.org/abs/2309.06180 (HTML opened 10 November 2026). OPT-13B bytes per token, reservation waste, block allocation at a summary level.

This article was originally published on 10 November 2026.