LLMs can now read and answer questions about long documents, but the memory required for inference grows with the context window. One reason GPU memory can suddenly run out when the model weights have not changed—but concurrency rises or a long document is supplied—is the KV cache.

The KV cache was originally introduced to speed up inference. It stores part of the computation for tokens the model has already read and reuses those results when generating the next token. Once tens of thousands of tokens must be retained, however, storing and reading this cache becomes a new bottleneck.

What makes the field interesting is that the label KV-cache optimization covers several quite different lines of research:

• Reducing unused space in GPU memory
• Designing model architectures that create fewer KV heads in the first place
• Retaining only important tokens
• Quantizing KV data to 2–4 bits
• Serving systems that place KV data across GPUs, CPUs, SSDs, and multiple servers

This post begins with what the KV cache is, then traces how representative studies reduce different aspects of its cost. Finally, it asks whether a higher compression ratio always leads to faster, better serving.

LLMs Generate an Answer One Token at a Time

A typical decoder-only LLM is an autoregressive model. Suppose it is generating this sentence:

The KV cache makes LLM inference faster.

The model does not produce the entire sentence at once. It might first generate The, inspect the tokens so far to choose KV, then look at The KV to select the next token, repeating this process until the sentence is complete.

The probability of an output of length $T$ factorizes as:

$$p(x_{1:T}) = \prod_{t=1}^{T} p(x_t \mid x_{<t}).$$

Because $x_t$ must be known before $x_{t+1}$ can be computed, generation is inherently sequential.

Self-attention in each Transformer layer constructs queries, keys, and values from its input:

$$Q = XW_Q, \qquad K = XW_K, \qquad V = XW_V.$$

A query represents the information needed at the current position. A key is more like an index indicating what information each earlier position contains. When the inner product of a query and key is large, the model retrieves more of that position's value.

$$\operatorname{Attention}(Q,K,V) = \operatorname{softmax}\left(\frac{QK^\top}{\sqrt{d}}\right)V.$$

To generate one new token, the query at the new position looks up every previous key and uses those scores to compute a weighted sum of the values.

The keys and values of past tokens do not change at later generation steps. Under causal attention, an earlier token cannot see a token that is added after it.

There is therefore no reason to recompute keys and values that have already been calculated.

What Does the KV Cache Store?

The KV cache stores the keys and values of previous tokens at every Transformer layer.

Suppose the prompt contains 1,000 tokens and the model has generated one new token. Producing the next token requires:

• Keys and values for the previous 1,001 tokens, read from the cache
• A new query, key, and value for the latest token, computed in the current forward pass
• The new key and value, appended to the end of the cache

Without a cache, every generation step would have to pass the entire, ever-growing prefix through the model again. With a KV cache, the model does not repeat the earlier tokens' K/V projections or their computations in preceding layers.

This has made the KV cache a standard component of modern LLM serving. Eliminating duplicate computation does not eliminate all generation cost, however. A new query must still compute scores against every past key and read the corresponding values. As context grows, so does the amount of cache read to generate each token.

Prefill and Decode Must Be Distinguished

LLM inference is usually divided into two phases.

Prefill

Prefill processes the user's entire prompt at once and creates the initial KV cache. Multiple prompt tokens can be handled through parallel matrix multiplications, so this phase makes relatively good use of GPU compute. Long prompts also require substantial attention computation, making prefill sensitive to both compute and memory.

Decode

Decode generates the output one token at a time. For a single request, the axis of newly computed tokens is nearly always one, so the matrix multiplications are small. Each layer repeatedly reads the model weights and the entire KV cache accumulated so far. With a small batch or long context, decode is therefore often limited by memory bandwidth rather than arithmetic throughput.

That is why a KV-cache compression paper must be read carefully: does it measure prefill time, time to first token (TTFT), inter-token latency (ITL), or total throughput?

How Large Does the KV Cache Get?

The approximate size of a conventional KV cache is:

$$M_{\mathrm{KV}} = 2 \times B \times L \times T \times H_{\mathrm{KV}} \times d_{\mathrm{head}} \times s.$$

The symbols mean:

• $2$: the key and value tensors
• $B$: batch size, or the number of concurrent requests
• $L$: number of Transformer layers
• $T$: number of tokens seen so far
• $H_{\mathrm{KV}}$: number of KV heads
• $d_{\mathrm{head}}$: dimension of one head
• $s$: bytes per element; FP16 and BF16 use 2 bytes

For example, Llama 3.1 8B has 32 layers, 8 KV heads, and a head dimension of 128. Storing 32K tokens in FP16 consumes about 4 GiB for the KV cache of a single request:

$$2 \times 32 \times 32768 \times 8 \times 128 \times 2 = 4\ \mathrm{GiB}.$$

Four concurrent requests require roughly 16 GiB. Model weights, activations, temporary buffers, and memory-allocator overhead must be stored separately. Extending the context to 128K or increasing the batch size grows the KV cache linearly.

This creates two distinct problems:

1. Memory capacity limits how many requests fit on one GPU.
2. Memory bandwidth determines how long reading past KV data takes for every next token.

Solving the first problem allows a larger batch. The cost of reading the model weights once can then be shared by more requests. Consequently, throughput gains in KV-cache reduction papers often come not from the compression operation itself being faster, but because more requests fit on the same GPU.

One further distinction is important. FlashAttention changes the order of operations so that the full attention-score matrix is never written to high-bandwidth memory. It reduces I/O for intermediate attention tensors, but does not eliminate the past keys and values that must persist throughout generation. FlashAttention and KV-cache compression target different parts of the same broad memory problem and can be used together.

Five Directions for KV-Cache Optimization

The research landscape becomes clearer if we view the KV-cache tensor as $[\text{Layer},\text{Token},\text{KV head},\text{Dimension}]$.

Some methods leave every tensor value untouched and merely manage storage more efficiently. Others reduce the token axis, while still others reduce the number of heads, dimensions, or bits. A practical serving system usually combines several of these methods rather than choosing just one.

1. Keep the Cache Values but Manage Memory Better

PagedAttention: Do Not Reserve Contiguous Memory in Advance

The final output length of a request is unknown until generation finishes. A naive implementation reserves a large contiguous memory region for each request or copies the cache to a larger region whenever it runs out of space. This produces unused gaps and reallocation overhead.

PagedAttention borrows an idea from operating-system virtual memory. It divides the KV cache into fixed-size blocks and uses a block table to map the logical token order to physical locations in GPU memory. Blocks can be added only when a request grows, and several sequences can easily reference the same prefix.

PagedAttention neither approximates nor discards keys and values. It is a system-level optimization that reduces memory fragmentation without changing model quality. This is why it is known as a core component behind vLLM's high throughput.

Prefix Cache: Compute a Repeated Prompt Only Once

The same prefix can recur across requests: a system prompt, few-shot examples, or a long shared document. There is no need to repeat prefill and store another identical copy of the KV data for every request.

Prompt Cache proposes precomputing and sharing the KV data of reusable prompt modules. PagedAttention-based serving can likewise use automatic prefix caching to reuse blocks for identical token prefixes.

Similar strings cannot automatically share a cache. The token sequence, model, positional treatment, and other execution conditions must match those used when the cache was created. Unlike the ordinary KV cache, which reuses the past within one request, a prefix cache reuses the identical beginning across requests.

Mooncake: Treat the KV Cache as a Distributed Store

Once a model and its serving cluster span multiple nodes, the KV cache is no longer just a tensor inside one GPU. A prefill worker's cache must be transferred to a decode worker. Caches evicted from GPU memory may be kept in CPU DRAM or on SSD and later returned to the device that needs them.

Mooncake designs a KV-centric distributed storage architecture for disaggregated prefill and decode, spanning CPUs, DRAM, SSDs, and RDMA networks. For this family of systems, where the cache is created, how it moves, how frequently it is reused, and available network bandwidth matter more than a compression ratio alone.

2. Make the Model Produce Less KV Data

Instead of compressing the cache at serving time, a model can be designed to create less KV data from the outset.

MQA and GQA: Share KV Across Query Heads

Standard multi-head attention (MHA) gives every query head its own key and value heads. A model with 32 query heads therefore also has 32 KV heads.

Multi-Query Attention (MQA) shares a single key head and value head across all query heads. This greatly reduces both the KV cache and the memory bandwidth needed to read it, though excessive sharing can affect quality.

Grouped-Query Attention (GQA) occupies the middle ground. It divides query heads into groups and assigns one KV head to each group. With 32 query heads and 8 KV heads, the cache is about one quarter the size of MHA with the same head dimension. GQA is common in recent LLMs because it offers a strong compromise between quality and serving cost.

MLA: Cache KV in a Lower-Dimensional Latent Space

Multi-head Latent Attention (MLA), introduced by DeepSeek-V2, compresses key and value information into a low-dimensional latent vector for caching. During decode, the required computation is absorbed into the weights or reconstructed from the latent representation before attention is performed.

Describing MLA simply as “turning KV into one vector” overlooks its treatment of RoPE and the separate key component. The actual design stores both a compressed KV latent and a key component carrying positional information. The core idea nevertheless remains: cache a far smaller latent representation instead of preserving a large K/V tensor for every head.

The Layer Axis Can Also Be Reduced

KV-cache size is proportional to the number of layers. YOCO divides the decoder into a self-decoder and cross-decoder and proposes caching KV only once. MiniCache exploits similarity between adjacent layers to merge caches in intermediate layers.

Architectural changes can produce large reductions, but they apply less broadly than post-training methods that modify only the cache of an already deployed model. One must determine whether new training or fine-tuning is required and whether existing inference engines support the attention variant.

3. Keep KV Only for Important Tokens

Not every token in a long context is equally important at every generation step. Recent conversation, sentences directly related to the question, and certain positions that consistently receive high attention often matter more. This observation can be used to reduce the token axis.

Scissorhands and H2O: Importance Tends to Persist

Scissorhands observes a persistence of importance: tokens that mattered in the past tend to remain important later. It removes less important KV entries based on this behavior.

H2O retains both heavy hitters—tokens with large cumulative attention scores—and recent tokens. Discarding everything merely because it is old can erase early instructions or central facts. Keeping only positions that have already received high attention denies new information a chance to prove useful. H2O preserves both groups for this reason.

StreamingLLM: Attention Sinks and a Recent Window

A simple sliding window retains only recent tokens, but its performance can become unstable during very long generation. StreamingLLM analyzes attention sinks, in which the earliest tokens receive large attention regardless of sentence meaning, and preserves both a few initial tokens and a recent window.

This is well suited to processing an endless stream with a fixed-size cache. It can be unfavorable, however, when the task must later retrieve a precise fact from the middle of the context. “Accepting a long input” is not the same as “remembering every piece of information in a long input.”

SnapKV: Use Attention Near the End of the Prompt to Select Earlier Tokens

SnapKV selects important positions earlier in the prompt using the attention pattern in an observation window at the prompt's end. It pools around selected positions to retain local context as well.

In the common arrangement where the question or instruction appears at the end of a prompt, the observation window can reveal which earlier regions the request needs. A limitation is that evicted KV entries cannot be recovered if the model's focus changes during generation or information becomes relevant only at a later reasoning step.

Quest: Select Pages to Read for This Query Instead of Discarding Them

Quest differs somewhat from permanently deleting token-level KV entries. It stores the cache in pages and has the attention kernel read only pages likely to be relevant to the current query.

Quest therefore reduces primarily memory I/O during attention. The full cache remains in storage, so a later query may choose different pages. Eviction that reduces cache capacity must be distinguished from sparse retrieval that reduces bandwidth.

Recent Work Looks Beyond Individual Tokens to Semantic Units

Per-token attention scores fluctuate across heads and layers, and removing isolated tokens can fragment sentence structure. ChunkKV retains KV data in semantically connected chunks. KVzip learns query-independent importance so that a compressed cache can be reused, while R-KV targets robust selection across multiple queries and long generation.

The field is moving from “delete tokens with low attention right now” toward “how do we retain information that may be needed again by future queries and later reasoning steps?”

4. Represent KV with Fewer Bits and Dimensions

Evicting a token removes its information completely. Quantization instead retains every token at lower numerical precision. Storing an FP16 element in 4 bits theoretically reduces its element storage to one quarter; 2 bits reduces it to one eighth.

But the KV cache is not an ordinary activation tensor. A small error in a key can pass through its inner product with a query and the softmax to change which position the model attends to. An error in a value is mixed directly into the attention output. The fact that the cache continually accumulates throughout generation also distinguishes it from weight quantization.

KIVI: Keys and Values Should Not Be Quantized the Same Way

KIVI observes that unusually large outliers repeatedly occur in particular key channels, while values do not exhibit the same clear pattern.

If a quantization scale is chosen independently for every token, outlier key channels can squeeze the remaining values of each token into a narrow range. Reflecting this asymmetry, KIVI performs 2-bit per-channel quantization for keys and per-token quantization for values. It keeps a small window of recent tokens in FP16 to reduce both quantize/dequantize overhead and error.

The broader lesson is not “K and V have the same shape, so use the same quantizer.” Their roles within attention and their statistical distributions must be considered separately.

KVQuant and GEAR: Handle Outliers and Residuals Separately

KVQuant jointly designs a nonuniform datatype matched to the KV distribution, per-channel key quantization, outlier separation, and a dense-and-sparse attention kernel. Merely lowering the bit width may not improve real speed because dequantization has a cost; designing the kernel alongside the representation is therefore important.

GEAR exploits the observation that the error left after quantization can be described by low-rank structure and sparse outliers. It adds low-rank and sparse corrections to a low-bit base tensor. The storage format becomes more complex, but remains smaller than preserving every value at high precision.

Palu: Project KV Heads into Lower Dimensions

Palu factorizes the key and value projection weights at low rank and caches the resulting low-dimensional latent representation. Its applicability to already trained MHA and GQA models distinguishes it from MLA as a model architecture.

Quantization reduces the bits-per-element axis, while low-rank techniques reduce the dimension axis. The two can be combined, but real latency must account for reconstruction and kernel support.

Where Does TurboQuant Fit?

Google Research's TurboQuant begins with a broader problem than a KV-cache-only quantizer. It addresses online quantization of high-dimensional vectors whose inner products must be preserved, presenting the attention KV cache as a primary application. The work appeared on arXiv in 2025 and was presented at ICLR 2026.

For this reason, taxonomies centered on token-eviction methods can easily overlook it. Yet it is a representative attempt to retain an extremely low-bit cache for long contexts while preserving attention inner products, so it belongs naturally in a discussion of KV-cache optimization.

TurboQuant's pipeline can be simplified as follows.

1. Spread Outliers with a Random Rotation

When large values are concentrated in a few vector coordinates, it is difficult to use the narrow range of a low-bit representation effectively. TurboQuant applies a structured random rotation that spreads the vector's energy more evenly across coordinates. Making the distribution more nearly Gaussian allows a single scalar quantizer to work effectively.

2. Store the Main Signal and Quantization Residual Separately

The main signal of the rotated vector is stored in $b-1$ bits with a scalar quantizer designed to minimize mean squared error. A 1-bit QJL (Quantized Johnson-Lindenstrauss) sketch corrects the residual left by quantization. The paper evaluates value-reconstruction error along with the bias and variance of estimated inner products between queries and keys.

3. 2.5 and 3.5 Bits Are Average Storage per Channel

The paper's 2.5 bits/channel and 3.5 bits/channel are not conventional integer data types. They are average storage costs after combining several components and group metadata. The paper describes 3.5 bits/channel as a quality-neutral regime and 2.5 bits/channel as a more aggressive setting with a small quality loss.

In the table, Llama 3.1 8B scores 50.06 on average with the full cache, 50.06 with TurboQuant at 3.5 bits, and 49.44 at 2.5 bits. This does not guarantee that 3.5 bits always matches FP16 for every model and task. It is the result for those models, datasets, and generation settings, and differences remain across individual LongBench tasks.

TurboQuant matters for more than reporting the smallest number. It uses a rotation to create a distribution that is easier to quantize, applies different representations to the main signal and residual, and designs around the statistical properties of the inner product that attention actually needs. It exemplifies the evolution of KV-cache quantization from a simple dtype conversion into an attention-aware numerical method.

OjaKV: Track Important Subspaces Online During Generation

The 2026 OjaKV method exploits the observation that KV-cache vectors occupy a low-dimensional subspace, but does not precompute a fixed low-rank basis. As tokens arrive, it updates the principal subspace online with Oja's algorithm and projects KV data into the current basis for storage.

The goal is to adapt to distribution shifts during generation without depending heavily on calibration data. It shows that recent research is expanding beyond “how many bits can we use?” to “how can we track the cache's time-varying structure online?”

Does a Higher Compression Ratio Make Serving Faster?

At this point, it may appear that making the KV cache smaller solves everything. At least three questions must be separated when interpreting results:

• How many bytes does the cache actually occupy?
• How quickly can the GPU read the compressed cache and compute attention?
• How do answer quality and output length change for the same prompt?

Throughput Rises When a Larger Batch Fits

The following KIVI experiment illustrates the relationship between memory savings and throughput.

At a small batch size, quantization and dequantization overhead can leave little benefit. If the smaller cache allows a larger batch, however, the cost of reading weights can be amortized across more requests and aggregate throughput increases. “How much faster is one request?” and “How many tokens per second can one GPU process?” are not the same question.

A Change in Output Length Can Distort Throughput Comparisons

Rethinking KV Cache Compression Techniques for LLMs reevaluates compression methods in a real serving stack and points out an important problem. Lossy compression can alter the model's generation distribution, causing different answer lengths for the same prompt.

If a compressed model ends its answer earlier, per-request latency may appear better merely because it generated less. If it begins repeating itself and produces a longer output, end-to-end latency worsens even when token-processing speed is unchanged. Reports should therefore include token throughput at a fixed output length, request latency under real stopping conditions, and answer quality.

The speed of a small PyTorch loop written for a paper can also differ from that of a serving engine with PagedAttention, continuous batching, and optimized kernels. We should ask whether the method theoretically reads fewer bytes and whether an actual kernel processes its compressed format directly.

An Average Benchmark Score Hides Which Capability Fails First

The 2026 ACL paper Pitfalls of KV Cache Compression for Instruction-Following in LLMs analyzes token-eviction methods from the perspective of instruction following.

Capabilities fail at different compression levels depending on the instruction type: formatting, length limits, language, punctuation, and so forth. A single average score can hide the fact that a particular instruction-following ability disappears first.

System prompts require even more caution.

The paper reports that aggressive compression can weaken adherence to system-prompt instructions or expose more of the system prompt in the answer. This does not mean every form of compression immediately leaks information. Behavior varies greatly by model and eviction policy. Before changing a production KV-cache policy, teams should evaluate instruction hierarchy, prompt leakage, tool use, and other real behaviors alongside ordinary long-context QA.

A Timeline of Representative Papers

The following is not an exhaustive bibliography. It selects papers that make shifts in the direction of the field easier to understand.

Surveys such as KV Cache Compression, But What Must We Give in Return? organize the field into token-level, model-level, and system-level optimization and provide a useful starting point for finding more individual papers.

What Should You Choose for Each Situation?

There is no single optimal technique. The right starting point depends on the bottleneck and the acceptable change in quality.

A cautious implementation sequence is:

1. Begin with lossless system optimizations such as PagedAttention and prefix caching.
2. If model choice is flexible, consider cache-friendly architectures such as GQA or MLA instead of MHA.
3. Profile whether memory capacity or bandwidth is the actual bottleneck.
4. Then choose quantization, token eviction, or sparse retrieval to match that bottleneck.
5. Evaluate the LongBench average together with real prompt distributions, output length, instruction following, and safety behavior.

Conclusion

The KV cache prevents an LLM from recomputing its entire past at every step. It is an unequivocal benefit in a short conversation, but as context and batch size grow, the cost of storing and reading keys and values at every layer becomes as important as the model weights themselves.

Research in this area ultimately answers one of the following questions:

What must be remembered, at what precision, where should that memory live, and when should it be read?

PagedAttention reduces wasted storage; GQA and MLA make the cache smaller from the outset. H2O and SnapKV choose important tokens, while Quest reads only the pages currently needed. KIVI and TurboQuant try to preserve every token using fewer bits. Mooncake extends the cache into a storage system spanning multiple devices and nodes.

As recent reevaluations show, a high compression ratio is not sufficient. We must ask whether a system merely appears faster because it produces shorter answers, whether specific instruction-following abilities fail first, and whether real GPU kernels process the compressed format efficiently.

The goal of KV-cache optimization is not to make the cache as small as possible. It is to preserve the history the model needs to choose its next token at the lowest cost allowed by the quality and latency constraints.