How a Transformer Really Works: Attention, the KV Cache, and Why Inference Eats Memory

In Neural Networks From Zero we built the machine from one neuron up and learned to size it: a 70B model is 140 GB of weights. But running it needs more memory than that — a second pool that starts near zero and grows with every token of the conversation, until one long request can cost ~10 GB on its own. Why does one user’s conversation cost 10 GB? Why do people obsess over inference memory? Why does long context get so expensive?

You can’t answer any of that from the outside. The cost is a consequence of how a transformer actually works — specifically, how it generates text one token at a time. So this post opens the box. By the end you’ll understand the architecture, what Q/K/V mean, and — the payoff — exactly why memory grows with every token you generate. That growth is the root cause the whole optimization field is built around.

The map: what’s actually stacked inside

Strip away the mystique and a transformer is a sandwich you already half-understand. Text comes in, gets turned into vectors (the numbers we talked about), flows through the same block repeated N times, and comes out as a prediction for the next token.

“Layer 1, Layer 2, … Layer N” just means this identical block, stacked N times — 80 times for Llama 3 70B. And each block has exactly two parts:

Hold onto that one-line distinction, because it’s the soul of the architecture:

Self-attention lets tokens talk to each other (a word looks at its context). The feed-forward network then lets each token think on its own. Mix, then think. Repeated N times.

And here’s the reassuring part: every box with _proj in its name is just a linear layer — a weight matrix, exactly the kind you counted in the last post. Self-attention is 4 of them; the FFN is 3. That’s all the learned weights are. The famous “12 × d² parameters per layer” is literally these seven matrices added up (≈4·d² for attention + ≈8·d² for the FFN). Nothing new — just assembly.

So the only new idea to understand is self-attention. Let’s earn it.

Why attention has to exist

A word’s meaning depends on its neighbors. “fox” in “the quick brown fox” is an animal; “fox” in “Fox News anchor” is a brand. Same token, same starting embedding — different meaning, and the only difference is the surrounding words.

So a token’s raw embedding can’t be its final representation. Each token needs a way to look at the other tokens and pull in the relevant ones to sharpen its own meaning. That act of looking-and-pulling is attention. The question is how a token decides which others are relevant — and that’s what Q, K, and V are for.

Q, K, V — what they actually mean

This is the part that never clicks from formulas alone, so here’s the analogy that makes it click, then the concrete version.

Think of a library search. You walk in with a question. Every book has a short label on its spine advertising its topic. You compare your question against every spine label to judge how relevant each book is, then you actually read the contents of the books — paying most attention to the most relevant ones — and synthesize an answer.

That’s attention, exactly:

Every token produces all three, by passing its own vector through three different weight matrices — q_proj, k_proj, v_proj. Same input vector, three different learned “lenses”:

Attention in three steps

Now watch one token — “fox” — update itself by attending over the four tokens “The quick brown fox”. “fox” is the 4th token, so it has query Q4.

Step 1 — score. Compare Q4 against every token’s Key using a dot product. A dot product is just a similarity number: high when two vectors point the same way. This asks “how relevant is each token to what fox wants to know?”

  Q4 · K1  → 0.2     ("The"    — barely relevant)
  Q4 · K2  → 3.1     ("quick"  — very relevant)
  Q4 · K3  → 1.4     ("brown"  — somewhat)
  Q4 · K4  → 2.7     ("fox"    — itself, relevant)

Step 2 — normalize with softmax. Raw scores aren’t weights; softmax turns them into fractions that sum to 1 (and exaggerates the big ones):

  raw:     [0.2, 3.1, 1.4, 2.7]
  softmax: [.03, .52, .10, .35]   ← attention weights, add to 1.0

So “fox” decides: pay 52% attention to “quick”, 35% to itself, 10% to “brown”, 3% to “The”.

Step 3 — blend the Values. Take each token’s Value vector, scale it by that weight, and add them up. Then one more matrix, o_proj, mixes the result — and that’s the new, context-aware vector for “fox”:

  new_fox = .03·V1 + .52·V2 + .10·V3 + .35·V4   →  o_proj  →  updated "fox"

That’s it — that’s attention. A weighted average of everyone’s Values, where the weights come from how well your Query matches their Keys. Do this for every token, in every block, and information flows across the sentence until each token’s vector encodes not just itself but its whole relevant context.

Don’t take my word for the numbers — play with them. See it in GPT-2 example below, token by token. Below is one attention head of GPT-2 (117M) running on our sentence. Pick a token and watch what it attends to as it comes in:

Each token attends over itself and the tokens before it — attention is causal, it can’t see the future — weighting them by relevance: fox leans on brown and quick, the words describing it. Those weights then blend each token’s Value vector into fox’s updated representation, exactly the mechanism from the diagram above. (This is just one of GPT-2’s ~144 heads; others specialize differently — some track the previous token, some fixate on the first token — and only together do they capture grammar and meaning.)

The root cause: generating text one token at a time

Here’s where memory enters, and it’s the whole point. An LLM does not write a sentence at once. It’s autoregressive: it generates one token, appends it to the input, and runs the entire model again to get the next one. Like predictive text that feeds its own output back in, forever.

  "The quick brown"        → model → "fox"
  "The quick brown fox"    → model → "jumps"
  "The quick brown fox jumps" → model → "over"
  ...

Now connect that to attention. To generate the next token, its Query must be scored against the Keys and Values of every token so far. Token 1000 attends over 999 predecessors. Which raises the obvious, expensive question: at each step, do we recompute K and V for the entire history?

We could — and it would be catastrophic. Recomputing all past K/V every step is O(n²) work over a sequence. So instead we do the one optimization that makes generation viable at all: we compute each token’s K and V once, and cache them. New token arrives → compute its K and V → append to the cache → done. The Keys and Values of old tokens never change, so there’s no reason to ever recompute them.

And _there_** is your root cause.** The cache grows by one K/V pair per token, per layer, per head — and it never shrinks during a request, because every future token might still want to attend to it. That’s the whole answer to “why does memory keep getting bigger over each iteration”:

• Weights are constant — 140 GB whether you generate 1 token or 100,000. Loaded once, shared by everyone.
• The KV cache is the opposite — it starts near zero and grows linearly with every token generated, and it’s private to each request.

Put real numbers on it for Llama 3 70B: 2 (K,V) × 80 layers × 8 KV-heads × 128 head-dim × 2 bytes ≈ 320 KB per token. At 32K tokens → 10 GB, for one conversation. Now you know why: it’s 320 KB accumulated 32,768 times, one autoregressive step at a time.

Here’s that formula as a live calculator — pick a model, drag the context length and the number of concurrent users, and watch the KV cache fill the GPU against the fixed weights (it’s the Red Hat slide that started this, except now you can break it yourself):

Pick Llama 3 70B at 32K context and one request is ~10 GB; drag the user count up and the cache races past the weights until it overruns the node. Switch to FP8 and every number halves. That single bar is why prefix caching, paged-KV, GQA, and quantized-KV all exist.

Drag it around and the scaling speaks for itself. Here is the same model at the context lengths people actually deploy:

Sit with that last row. A single 128K-context request needs 40 GB of KV cache — nearly a third of the model’s entire 140 GB of weights — for one user’s conversation. Now serve 10 such users at once: 400+ GB of KV cache alone, on top of the model. That is more than the entire memory of a 4×80 GB node. This — not the math, not the weights — is why managing the KV cache is the single biggest job of a production inference server.

Where does all this live? The GPU memory hierarchy

We’ve been saying “GPU memory” as if it’s one thing. It isn’t — and its structure is the last piece that makes why data movement dominates finally click. A GPU has three tiers, and they obey the same law as every memory system you already know, from a database’s buffer pool to your CPU’s caches: the closer to the compute, the faster — and the smaller.

If you know CPU caches, you already know this shape: SRAM is the L1 cache — tiny, on-chip, sitting right next to the compute units (the tensor cores) and feeding them at a staggering ~19 TB/s. HBM is main RAM — gigabytes, but an order of magnitude slower. Host DRAM is the far-off disk, reached over a thin ~12 GB/s cable. Now place our two tenants:

• Model weights (140 GB) load into HBM once at startup and stay there for the server’s life.
• The KV cache also lives in HBM, growing per request — which is why HBM capacity, not compute, caps how many users you can serve.
• Every forward pass, small chunks of weights and KV are pulled HBM → SRAM for the tensor cores to work on, then results are written back.

And there’s the physical reason decode is memory-bound: the tensor cores are so fast they finish and sit idle, waiting for the next chunk to arrive over the HBM→SRAM path. The bottleneck isn’t the arithmetic — it’s the data movement.

This also finally explains why we cache K and V but not Q. Tensors come in two kinds. Persistent ones — the weights and the KV cache — stay in HBM because they’re needed again and again. Transient ones — a token’s Q, its freshly-computed K and V, the attention output, the FFN output — are born in SRAM, used immediately, and discarded. Q is transient (used once, the moment its token does the looking); K and V get promoted into the persistent cache precisely because every future token will still need them.

That leaves one sentence the entire optimization field hangs on: every technique you’ll meet is one of three moves — move less data, move it more efficiently, or manage the memory better. Quantization shrinks the bytes; paged KV manages the cache; FlashAttention keeps more of the work inside SRAM. Same three levers, over and over.

Why this is the hinge of the entire field

Step back and the two costs you now understand explain nearly every inference technique in existence:

• Memory-bound decode — each step drags all 140 GB of weights through the chip to compute one token. Barely any math per byte. That’s why decode is slow and why batching many users’ tokens together is the throughput lever. (A dedicated post on this — the roofline model — is coming.)
• The growing KV cache — caps how many users fit in memory at once, and makes long context expensive. That’s why paged KV cache, quantized KV, GQA, MLA, prefix sharing all exist. Every one is a strategy to store, shrink, or reuse this cache.

This is why memory dominates every conversation about inference, and the reason is structural: an autoregressive transformer is a machine that re-reads all of its weights and its entire growing memory to produce one token at a time. The whole field of inference optimization is the collected set of tricks for making that bearable.

What comes next

Everything downstream builds on the two costs in this post — and the highest-leverage trick attacks the bigger one, the KV cache. Under naive allocation, most of a request’s reserved cache sits unused: you set aside room for the full context length, but a typical request never fills it. Paged KV cache — the idea behind vLLM’s PagedAttention — reclaims that waste with a technique borrowed straight from operating-system memory paging. That’s where this series goes next.