Table of Contents

If you’ve used ChatGPT or any Transformer-based language model, KV Cache is why the response speed is tolerable. Without it, the model would need to recompute attention over all previous tokens every time it generates a new word — for long text, that’s quadratic compute blowing up fast. This article explains what KV Cache is, why it works, and how modern LLMs continue to push its limits within memory constraints.

TL;DR

KV Cache stores the Key and Value matrices for all previously seen tokens during Transformer self-attention. When generating each new token, only the current token’s Q needs to be computed; the model then reads from the cache to perform attention. This reduces the time complexity of autoregressive generation from O(n²) to O(n) per step — at the cost of memory growing linearly with sequence length, which becomes a GPU memory bottleneck for long-context inference.

Design Philosophy

Understanding KV Cache requires understanding what self-attention actually does.

For each token in the input sequence, self-attention computes three vectors:

  • Query (Q): “What am I looking for?”
  • Key (K): “What can I be found for?”
  • Value (V): “If someone finds me, what do they get?”

The attention score is the dot product of Q with each K, softmaxed, then used to weight-sum over V:

Attention(Q, K, V) = softmax(QK^T / √d_k) × V

In autoregressive generation, the model generates one token at a time. When generating token n, its Q must dot-product with the K of all n-1 previous tokens. Without caching, every new token forces a full recomputation of all previous K and V matrices — that’s where the O(n²) problem comes from.

The KV Cache solution is straightforward: keep the computed K and V around. Each new token only computes its own Q, K, V, appends the new K and V to the cache, then uses Q to attend over the full cache.

Core Concepts

Cache Structure

At token t generation time:
KV Cache = { K_1, K_2, ..., K_{t-1},
             V_1, V_2, ..., V_{t-1} }

Steps:
1. Compute Q_t, K_t, V_t for current token
2. Append K_t to cache → K_1...K_t
3. Append V_t to cache → V_1...V_t
4. Attention(Q_t, [K_1...K_t], [V_1...V_t]) → output

Each token’s computation is O(n) (dot-producting against all K/V in cache). The total computation for a sequence of length n is O(n²), but amortized per token it’s O(n), avoiding redundant recomputation.

Memory Footprint

KV Cache memory usage is:

KV Cache size = 2 × n_layers × n_heads × d_head × seq_len × bytes_per_element

For Llama 3.1 8B (32 layers, 32 KQ heads, 128 dimensions, float16):

  • Per token: 2 × 32 × 32 × 128 × 2 bytes = 524,288 bytes ≈ 0.5 MB
  • Cache for 128K context: 128K × 0.5 MB = 64 GB

This is why KV Cache fills GPU memory for long-context inference. An A100 (80GB VRAM) running a 128K-context 8B model uses 64GB just for the cache — barely any room left for the model weights.

Prefill vs. Decode Phases

LLM inference has two distinct phases:

Prefill: The entire input prompt is processed at once, building the initial KV Cache. This phase is compute-intensive; GPU utilization is high.

Decode: Generates one token at a time, reading historical KV from cache. Compute demand is relatively low, but it’s bottlenecked by memory bandwidth (reading large caches from GPU HBM). For long sequences, the decode phase bottleneck is memory bandwidth, not compute.

Comparison with Alternatives

ApproachProsConsUse Case
Full KV CacheSimplest, fastestMemory grows linearly with sequenceShort to medium sequences
Multi-Query Attention (MQA)KV cache shrinks by n_headsRequires retrainingLatency-sensitive inference
Grouped-Query Attention (GQA)Balances MHA quality with MQA efficiencyAlso requires retrainingLlama 3, Mistral, modern models
Multi-head Latent Attention (MLA)Cache compressed to low-dim latent spaceExtra projection at inference timeDeepSeek V3
Sliding Window AttentionO(w×n) instead of O(n²)Loses long-range dependenciesSome Mistral layers
KV Cache EvictionCaps memory usageMay drop important tokensMorphKV and research

GQA (Grouped-Query Attention) is the most widely deployed optimization today. In Llama 3.1, the 32 Query heads are split into 8 groups that each share a single KV pair — reducing KV Cache size to 1/4 with minimal quality loss.

MLA (Multi-head Latent Attention) is DeepSeek V2/V3’s innovation. It projects K and V into a low-dimensional latent vector, caches the compressed representation, and expands at inference time. In theory this can reduce cache size by 5–13×, but implementation complexity is higher.

When KV Cache Helps (and When It Doesn’t)

KV Cache at its most effective:

  • Long conversations (multi-turn Q&A where each turn includes full history)
  • RAG pipelines (analyzing documents after retrieval)
  • Batch inference where multiple requests share a prefix (prefix caching optimization applies)

Where KV Cache hits its limits:

  • Extremely long text (100K+ tokens): cache alone exceeds GPU memory
  • High-concurrency serving (many simultaneous users): each session needs its own cache
  • Edge deployment (extremely memory-constrained): cache size requirements hard to satisfy

The Bottom Line

KV Cache is the foundational technology that makes modern LLM inference viable. Without it, every ChatGPT response would be orders of magnitude slower. But it’s also the key bottleneck limiting long-context LLM deployment at scale.

The research direction in 2025 focuses on three areas: compression (GQA, MLA), eviction (intelligently dropping unimportant token caches), and distributed caching (PagedAttention, vLLM’s cache management). Progress in these areas directly determines the cost and latency envelope of LLM serving.

References

Ask this article

Answers come from this article only. Click any prompt below or open the chat at the bottom right.

🇺🇸 English

If you've ever chatted with ChatGPT, or really any Transformer-based language model, and thought, "huh, that's actually pretty snappy" — there's one piece of engineering you can thank for that. It's called the KV Cache. And without it, every response you get would be, quite literally, orders of magnitude slower. So let's talk about what it is, why it works, and why it's simultaneously the hero and the headache of modern AI inference.

Here's the one-sentence version to hold onto. When a language model generates text, it produces one word at a time, and each new word has to "pay attention" to every word that came before it. Without any cleverness, that means recomputing the entire history from scratch on every single step. That's quadratic — the work explodes as your text gets longer. The KV Cache turns that quadratic nightmare into something linear per step. It's the reason long-form generation is fast enough to actually use.

To really get why it works, we need to peek under the hood at what "attention" is doing.

For every word — every token — in a sequence, the model computes three things. Think of them as a little dating-profile metaphor. First, the Query: "What am I looking for?" Second, the Key: "What can I be found for?" And third, the Value: "Okay, if you found me, here's what you actually get." The model takes a token's Query, compares it against the Keys of all the other tokens to see which ones are relevant, and then pulls a weighted blend of their Values. That blend is the token's understanding of its context.

Now here's the crucial detail about how these models generate. They're autoregressive — one token at a time, left to right. So when the model is producing token number one hundred, that token's Query needs to compare against the Keys of all ninety-nine tokens before it. And here's the wasteful part: those ninety-nine Keys and Values? They don't change. Token fifty's Key is the same whether you're generating token fifty-one or token five thousand. But naively, the model recomputes all of them, every single step. That's the source of the O(n²) blowup.

And the fix is almost embarrassingly simple. Just... keep them around. Compute a token's Key and Value once, stash them in a cache, and never recompute them. When a new token arrives, you only compute its own Query, Key, and Value. You append its Key and Value to the growing cache, and then let its Query attend over that whole cache. That's it. That's the KV Cache.

So the mechanics, step by step: a new token comes in. You compute its Q, K, and V. You tack the new K and V onto the end of your stored cache. Then the current token's Query does its attention over the full accumulated set of Keys and Values, and out pops the output. Each step is now proportional to the length of the sequence, not the length squared. You've traded redundant computation for a bit of memory. And that trade is the whole ballgame.

But — and this is the big "but" — that memory cost is not free, and it grows. Let's make it concrete with Llama 3.1, the 8-billion-parameter model. It has thirty-two layers, thirty-two attention heads, a head dimension of 128, running in 16-bit precision. Crunch the numbers, and every single token you cache costs about half a megabyte. Sounds tiny. But now imagine a 128,000-token context — the kind of long document people love to feed these models. Half a megabyte times 128,000 tokens lands you at sixty-four gigabytes.

Sixty-four gigabytes. Just for the cache. Take an A100 GPU with eighty gigs of memory, and your KV Cache alone has eaten sixty-four of them, leaving barely enough room for the model's own weights. So this elegant little optimization that made everything fast? At long context lengths, it becomes the thing that's suffocating your GPU.

There's another wrinkle worth knowing, and it explains a lot about how inference actually behaves. Generation happens in two distinct phases. The first is called prefill — that's when the model ingests your entire prompt at once and builds up the initial cache. This phase is compute-heavy; the GPU is working hard and running hot. The second phase is decode — that's the token-by-token generation, where the model is mostly just reading from that big cache. And here's the counterintuitive part: decode isn't limited by raw compute. It's limited by memory bandwidth — how fast you can shuttle that enormous cache in and out of the GPU's high-speed memory. For long sequences, your bottleneck isn't the math. It's the memory pipe.

So naturally, a whole ecosystem of tricks has grown up to fight the memory problem. Let me walk you through the landscape.

The baseline is the full KV Cache — simplest, fastest, but memory scales straight up with sequence length. Great for short and medium text, painful for long.

Then there's Multi-Query Attention, MQA. The idea: instead of every attention head keeping its own Keys and Values, they all share one set. That shrinks the cache dramatically. The catch is you have to retrain the model to do it, and sharing everything can cost you some quality.

The sweet spot most modern models have landed on is Grouped-Query Attention, GQA. It's the compromise child of the two extremes. In Llama 3.1, for instance, those thirty-two Query heads get bundled into eight groups, and each group shares a single Key-Value pair. The result: your cache shrinks to a quarter of its size, and the quality hit is almost nothing. That's why GQA is everywhere now — Llama 3, Mistral, the modern lineup.

DeepSeek went a different, more ambitious route with something called Multi-head Latent Attention, MLA. Instead of sharing Keys and Values, it compresses them — projecting them down into a small low-dimensional latent vector, caching that compact version, and then expanding it back out when needed. On paper it can cut cache size by five to thirteen times. The price is more implementation complexity and a bit of extra work at inference time.

A few others round out the toolbox. Sliding Window Attention lets each token only look at a fixed window of recent tokens instead of the whole history — cheap, but you lose long-range memory, so Mistral only uses it in some layers. And KV Cache Eviction takes a more surgical approach: cap the memory, and when it fills up, intelligently throw away the least important tokens' cache entries. The risk, obviously, is guessing wrong and dropping something that mattered.

So when does all of this really earn its keep? The KV Cache shines in long multi-turn conversations, where every new turn drags along the full history. It shines in RAG pipelines, where you retrieve a big document and then reason over it. And it shines in batch serving where many requests share a common prefix — you can cache that shared part once.

And where does it hit a wall? Extremely long text, past a hundred thousand tokens, where the cache alone blows past your GPU memory. High-concurrency serving, where every simultaneous user needs their own private cache and it all adds up fast. And edge deployment, where you're so memory-starved the cache simply won't fit.

So let me leave you with the three things worth remembering.

First: the KV Cache is foundational. It's not a nice-to-have optimization — it's the reason real-time LLM inference exists at all. It converts quadratic recomputation into linear reading by simply remembering the Keys and Values it already calculated.

Second: its strength is also its curse. The very thing that buys you speed — storing all that history — is what devours GPU memory and caps how long a context you can practically serve. Memory, not compute, is often the real ceiling.

And third: the frontier of research is all about beating that memory tax, along three fronts — compressing the cache with tricks like GQA and MLA, evicting the parts you don't need, and managing it more cleverly across memory with systems like PagedAttention and vLLM. Progress on those three fronts is, quite directly, what decides how cheap and how fast your AI can be. So the next time a model answers you instantly over a giant document, you'll know exactly what's quietly working overtime behind the scenes.

🇹🇼 中文

如果你用過任何一個基於 Transformer 的語言模型，其實你一直都在享受 KV Cache 帶來的好處。它的 Cache 唸起來跟「錢」的 Cash 一樣——而它確實也跟錢脫不了關係，因為它花的是 GPU 記憶體，也就是成本。今天我們就來把 KV Cache 講清楚：它是什麼、為什麼會把你以為很大的倉庫直接塞爆，還有業界為了救這個倉庫，想出了哪些辦法。

先快速回顧一下，語言模型到底怎麼生成文字。本質上就是文字接龍。你先給一段 prompt，模型吐出一個 token，然後把這個 token 接回輸入、再吐下一個，一直到遇到結束符號為止。

這整個過程可以拆成兩段。第一段叫 Prefill，一次把一整段很長的輸入、把 prompt 全部算過一遍。第二段叫 Decode，接下來就是一個一個 token 慢慢往外吐。

在 Prefill 階段，假設輸入有三個 token，每個 token 都會算出自己的 Q、K、V，然後拿去算 attention。這裡有個關鍵——這三個 token 算各自輸出的動作，是可以完全平行運算的。

好，接下來就是 KV Cache 這個核心概念了，其實非常簡單。算完之後，我們把 K 跟 V 存下來，把 Q 直接丟掉。這個「存下 K 跟 V」的動作，就是 KV Cache。

為什麼可以丟掉 Q、只留 K 和 V？看 Decode 階段就懂了。前三個 token 進來後，模型生出第四個 token，這個新 token 只需要算它自己的 Q4、K4、V4。它的 Q4 去跟存好的 K1 到 K3、加上新的 K4 算 attention，再對 V1 到 V4 做加權總和，就得到輸出。前面那些 token 的 K 和 V，完全不用重算。想想看，如果每次都要把新 token 接著前面所有 token 重新丟進模型、重算一次 K 跟 V，那實在太浪費時間了。所以我們就直接讀快取。

概念就這麼簡單。但實作上，會撞到一個巨大的問題。

你原本以為 GPU 記憶體這個倉庫很大、放什麼都行，但 KV Cache 有本事把它直接塞爆，原因有兩個。第一，要存的 K 跟 V 非常多，每多一個 token，就要多存一組。序列一長，倉庫就滿了。第二，K、V 不只一組，因為我們通常做 Multi-Head Attention，有很多個 Query head，每一個都有自己對應的 Key 跟 Value。

我們拿 Gemma 2 來實際算一下。以 27B 這個版本為例，它有 46 層，我們先假設它是一般 attention，用 30 個 head、每個向量 128 維。算下來，每產生一個 token，要存的 K/V 大約是 736 KB，差不多 0.72 MB。一個 token 不到 1MB，聽起來還好對吧？但假設你用的是 A100，80GB 記憶體，這 80GB 也只夠存大約十一萬個 token。而現在我們對 context 的需求，動不動就遠超十萬。所以當你的輸入或輸出太長，你就會看到那句經典的 CUDA out of memory——那個你以為無比巨大的倉庫，是真的會被撐爆的。

為了讓倉庫撐久一點，業界發明了一大堆方法，我們一個一個看。

第一招，減少 K/V 的組數。既然被存下來的是 K 跟 V、Q 不會被存，那自然的想法就是：Q 可以多，但 K/V 要少。這裡有兩個做法。一個叫 Multi-Query Attention，MQA，所有的 Query 共用同一組 K 和 V，倉庫佔用大幅下降，但缺點是效果不太好。另一個叫 Grouped-Query Attention，GQA，它介於中間，仍然有好幾組 K/V，但每一組配給多個 Query 共用。比方說四個 Query 分成兩組，運作起來像是四個 Query 的 attention，卻只要存兩組 K/V。

這裡常被問到：為什麼是不同 Query 共享同一組 K/V，而不是反過來？答案就在 KV Cache——因為只有 K/V 會被存下來，讓它變少才有意義；Q 反正不進倉庫，多幾組無所謂。GQA 現在被廣泛採用，LLaMA、Gemma 這些知名模型裡都有它。

第二招更激進，叫 Multi-Head Latent Attention，MLA，DeepSeek 用的就是這一招。它的想法是：與其存好幾組 K/V，不如在中間塞一個 bottleneck，先把輸入壓成一個較低維的向量，我們叫它 C，倉庫裡就只存 C，之後再乘上不同的轉換展開。這是需要訓練的。

那你直覺上一定會擔心：算 attention 的時候，是不是得先把 C 解壓縮回一堆 K/V？如果要解壓縮，那算力根本沒省到啊。MLA 神奇的地方就在這裡——它不需要解壓縮，可以直接在壓縮的維度上算。原理其實是個很漂亮的數學技巧：本來 attention 要算 Q 乘上 W 乘上 C，我們可以把 Q 跟那個轉換矩陣先湊成一對、預先算好，再直接跟 C 做內積，結果完全等價，這不是近似，是數學上相等。好處是，壓縮 Query 這件事每個 Query 只要做一次；而 C 的數量是跟序列長度綁在一起的，你要解壓縮就得對一整串做，太花時間。加權總和那邊也是同樣的技巧，可以先在壓縮的 C 上做加總，最後只解壓縮一次。實務上，MLA 甚至能拿到比原本 Multi-Head Attention 稍微好一點的結果。

第三招，換個角度，不改組數，改成限制 attention 要看的長度。這叫 Sliding Window Attention，每次做 attention 不看整個序列，只看前面一個固定的 window，比方說 4096 的範圍。這樣 KV Cache 就有了固定上限。你可能會擔心看得太短，但因為 Transformer 是多層堆疊的，上層的 query 透過下層，還是能間接看到更前面的 token，只要層數夠深，等效的視野其實可以很大。這招曾經用在某個版本的 Mistral 7B。你也可以只把部分層換成 sliding window、其他層維持完整 attention，GPT OSS 就是這樣一層 sliding window、一層完整 attention 交錯著用。

第四招，Streaming LLM，還有一個很有意思的概念叫 attention sink。前面說的 Sliding Window，在輸入很長的時候常常會讓表現變差。Streaming LLM 發現一個極簡的救法：只要你的 attention 範圍裡，包含整個序列最開頭的那幾個 token，表現就穩了。而且這招連額外訓練都不用，只要在 inference 的時候，把最前面幾個 token 加進 sliding window，效果就大幅提升，能穩定處理訓練時從沒見過的超長輸入。

那為什麼開頭的 token 這麼重要？因為 attention 是強制的——所有 attention weight 加起來一定要等於 1，每個 query 無論如何都得把注意力放到某個地方。當一個 token 其實沒什麼好 attend 的時候，模型的預設行為就是把注意力全倒到第一個 token 上，這就是所謂的 attention sink。一旦你把第一個 token 拿掉，它整個世界就崩壞、不知道該怎麼辦；把第一個 token 還給它，行為立刻恢復正常。

第五招，最直接，把沒用的 K/V 從倉庫裡刪掉，也就是 pruning。2023 年有兩篇經典論文，Scissorhands 跟 H2O，都發現了非常類似的現象：多數 token 的 K/V，後來根本沒被 attend 過。你把 attention 視覺化就會看到，一大片區域的 token 從頭到尾沒人理它，存這些 K/V 只是白佔空間；反而有少數 token 會被反覆 attend，那些才值得留。所以這些方法的精神就像倉庫管理——一組 K/V 如果一直沒人來拿，過陣子就丟掉。Scissorhands 甚至能壓縮五倍，只留 20% 的 token，很多任務上表現還是差不多。不過後續也有文獻指出，碰到很難的任務，亂丟 K/V 還是會掉分，所以「怎麼 pruning 才最有效」，後來就衍生出滿坑滿谷的論文。

最後還有一個很酷的用法，跨對話的 KV Cache，也就是 prefix 共用。前面講的都是同一個對話內的快取，但它其實也能跨對話。假設模型先生成過「大家好我是大金」，存了快取，這時另一個人 prompt「大家好我是小金」，這兩句的前五個 token 一模一樣，那前五個 token 的 K/V 就能直接搬過去用。但要注意兩點：「大」跟「小」對應的 K/V 不同，所以整句不能互換；而且就算兩句都有「金」這個字，也不能把上面那個「金」的快取挪下來用。因為每個 token 算出來的表示，取決於它前面看過什麼，前面從「大」變「小」，representation 就變了。換句話說，只有擁有完全相同前綴的兩個序列，才能共用那段前綴的 K/V。

好，我們收個尾。今天有三個核心要點想讓你帶走。第一，KV Cache 本身一句話就講完了——把 K 跟 V 存下來、別重算；但它會隨著序列長度和 head 數量線性膨脹，這正是限制長文本 LLM 部署的核心瓶頸。第二，所有的優化，其實都在做同一件事：讓那個會被撐爆的倉庫撐久一點——MQA、GQA 減少組數，MLA 壓進 latent 空間，Sliding Window 跟 Streaming LLM 限制範圍，pruning 直接丟掉沒用的。第三，記住那個反直覺的 attention sink：模型沒東西可看的時候會死抓著第一個 token，所以保留開頭，往往就是穩定超長輸入的關鍵。下次你看到 CUDA out of memory，你就知道，這座倉庫背後，藏著多少工程師的巧思。

Tags

#kv-cache#llm#inference-optimization#transformer#ai#machine-learning

Related Articles

tech

RAG's Five Stages: From Pipeline to Reasoning Retrieval, and the Naive RAG on My Own Site

Over the past two years RAG evolved from a 'linear pipeline' to 'loop-based reasoning'. It maps cleanly to five stages: Naive, Advanced, Modular, Graph, Agentic. The real inflection point is control moving from pipeline to agent — a System 1 → System 2 shift. Looking back at engineer-news's own RAG stack, it's stuck at the Naive edge — so this post also lays out what to fix next.

#ai#llm#rag#graph-rag#agentic-rag#system-design#cloudflare