Attention Is All You Need: a deep dive

My last two experiments compared techniques on top of an existing model: prompting strategies, then fine-tuning approaches. This one steps back further, to the architecture underneath all of it. If you’ve never traced through how a transformer actually turns a sentence into an answer, this is that walkthrough, plus a look at how today’s models (BERT, GPT, T5, and the efficiency and Mixture-of-Experts designs since) changed the original 2017 design, and why.

Try the live dashboard or read the full write-up and code.

Why this paper exists

Before 2017, the best sequence models read a sentence one word at a time, in order, carrying a running summary forward as they went. That’s slow to train, since word 50 can’t be processed until words 1 through 49 have been, and information from early words tends to fade by the time it’s carried all the way to a much later one.

Vaswani et al., 2017 removed that step-by-step reading entirely and replaced it with attention: every word looks directly at every other word it needs, all at once, no matter how far apart they are. That’s the whole idea behind the title: attention turned out to be all you actually need.

flowchart LR
    Q["Query: what this word is looking for"] --> S["Compare against every word's Key"]
    K["Key: what each word offers"] --> S
    S --> Sm["Softmax: turn scores into weights that sum to 100%"]
    Sm --> W["Weighted mix of every word's Value"]
    V["Value: the content each word carries"] --> W
    W --> O["Output for this word"]

The core mechanism, in plain terms

Every word gets turned into three different versions of itself: a Query (what it’s looking for), a Key (what it has to offer other words), and a Value (the actual content it passes along if there’s a match). A word compares its Query against every other word’s Key to get a similarity score, turns those scores into a set of weights that add up to 100% (using a function called softmax), and then builds its output as a weighted mix of every word’s Value using those weights.

That’s it. That’s attention. Everything else in the architecture, splitting it into multiple parallel “heads” so different heads can specialize on different kinds of relationships, adding position information so word order isn’t lost, stacking many layers of this with shortcuts and rescaling to keep training stable, is built around that one core computation.

I traced this by hand on a tiny 3-word sentence, “cats chase mice,” with small hand-picked numbers instead of a trained model’s real numbers, just to see the arithmetic work end to end. One thing that stood out: chase, the verb, ended up splitting its attention almost evenly between cats and mice, its subject and object. That’s a coincidence of the specific numbers I picked, not proof of anything a trained model does, but it made the mechanism click in a way reading the formula alone hadn’t.

How the design evolved, and why

The original architecture is a full encoder-decoder, built for translation. Almost nothing since has walked away from attention itself. Instead, each later design kept the mechanism and changed either which half of the architecture it uses, or how attention gets computed to make it cheaper to run.

BERT (Devlin et al., 2018) kept only the encoder half, where every word can look at every other word in both directions, and trained it by hiding random words and asking it to guess them. Good for understanding text, not for generating it.

GPT kept only the decoder half, where each word can only look at earlier words, trained simply to predict the next word over and over. That one restriction, no peeking ahead, is what lets it generate text one word at a time. This is the lineage behind essentially every chat-style model today: GPT, Llama, Mistral, and (at the level of publicly known trends, since Anthropic hasn’t published Claude’s internals) Claude too.

T5 (Raffel et al., 2019) kept the whole original architecture and instead changed how every task gets framed, as plain text in, plain text out, so one model and one training recipe can handle translation, summarization, and classification alike.

A second wave of changes targets a cost none of the above deal with: during generation, a model has to keep every earlier word’s Key and Value in memory (the KV cache), and that gets expensive over long text. RoPE (Su et al., 2021) bakes position into attention through rotation instead of adding it once at the start. Grouped-query attention (Ainslie et al., 2023) lets multiple heads share a smaller number of cached Key/Value heads. FlashAttention (Dao et al., 2022) changes nothing about the math, only how it moves through GPU memory, and turns out to be a big part of why long-context models became practical at all.

And Mixture-of-Experts models (Shazeer et al., 2017) swap a model’s single feed-forward network for many smaller ones, with a small router picking just a few to run per word, so total capacity can grow much larger without every word paying the full cost. DeepSeek’s models (DeepSeek-V2, 2024; DeepSeekMoE, 2024) push both ideas further: compressing the KV cache into a shared latent vector, and splitting experts into many small routed ones plus a few shared ones every word always uses.

What I found worth sitting with

Splitting attention into multiple heads doesn’t cost anything extra in parameters. It sounds like it should, more heads, more computation, but the total dimension just gets divided across them instead of multiplied. The benefit isn’t more capacity; it’s that each head gets its own smaller, independent comparison to specialize with.

The other thing that stood out, once I laid the timeline out: almost every efficiency idea since 2017 (RoPE, grouped-query attention, multi-head latent attention, FlashAttention) targets the same bottleneck, the KV cache that autoregressive generation has to keep growing. None of them touch the core attention formula itself. The formula from 2017 is still, underneath all of it, exactly what’s running.

The full write-up has every diagram, the complete worked-example matrices (single-head and split across two heads), and an honest note on where Claude and Perplexity do and don’t fit into this story.

Read the details, or poke around the dashboard yourself.