Attention

To understand a sentence, a model has to figure out how the words relate. Read the sentence "the animal didn't cross the street because it was tired." What does "it" refer to? Almost certainly the animal, not the street, because streets do not get tired. You resolved that by letting "it" look back at the other words and deciding which one matters. Attention is how a transformer does the same thing.

Every word looks at every other word

For each word, attention asks a simple question: to understand this word, how much should I focus on each of the other words? The answer is a set of weights, one per word, showing how much focus each word gets. A high weight means "this other word is important for me right now," a low weight means "I can mostly ignore it."

Crucially, every word does this for every other word, in both directions and all at once. That is what lets a model link "it" to "animal" even though several words sit between them. Distance in the sentence does not matter; only the learned relationship does.

Reading the figure

Click a word to see its attention as links to the rest of the sentence. Thicker, brighter links mean stronger attention. Click "it" and the strongest link snaps back to "animal," with only a faint link to "street." That is the model resolving what "it" stands for. Try "cross" and you will see it lean on "street" and "animal," the words that say what is being crossed and who is crossing.

The weights here are hand-set to make the lesson clear. A real model is not told that "it" means "animal." It learns weights like these from data, and it computes them on the fly from each word using learned query, key, and value vectors, usually across many attention heads at once, each head watching for a different kind of relationship. In the grid above, each cell's shade is the relative strength of one word attending to another, brightest for the strongest link.

Why it matters

Attention is the core mechanism of the transformer, the architecture behind modern language models. Before attention, models read text strictly in order and tended to forget what came far earlier. Attention removed that limit: any word can draw on any other word directly, no matter how far apart, which is what let these models handle long passages and capture the relationships that make language mean something.

What the demo shows

Each point is a word of the example sentence. Drag to orbit, scroll to zoom, and click a word to draw its attention links to the others, weighted by strength. The "it" to "animal" link is the headline case, hand-set so the idea of attention resolving a reference is easy to see.

Query, key, and value

So far you have seen that words attend to each other with different strengths, but not where those strengths come from. The mechanism is query, key, and value. Each word starts with its embedding and produces three vectors from it: a Query, representing what this word is looking for, a Key, representing what it offers to other words, and a Value, the information it passes along if another word attends to it.

To decide how much one word attends to another, the model compares the first word's Query against the second word's Key. It does this with a dot product, which is just a single number measuring how well the two vectors line up. A bigger number means a closer match.

But raw match numbers are not weights yet. Two things happen to turn them into weights. First, each match is divided by the square root of the vector size. This scaling keeps the numbers from getting too large, which would make the next step too extreme. Then all the scaled matches go through softmax, which turns a list of numbers into a set of weights that are all positive and add up to 1. Softmax also exaggerates the gaps: a slightly higher match becomes a clearly higher weight, so the strongest match stands out.

This is the real formula from the transformer paper: attention weights are softmax of the Query-Key match, scaled by the square root of the key size.

Try it in the figure. Click "it" and watch where its attention goes. Its strongest weight lands on "animal," which is what "it" refers to in the sentence. The model has no grammar rules telling it this. It falls out of the Query and Key vectors and the softmax over their matches.

The attention weights here are the real computation: a Query-Key dot product, scaled by the square root of the vector size, then softmax, exactly the transformer paper's formula. What is hand-authored is the Query, Key, and Value vectors themselves. A real model learns those from data over billions of words; here they are set by hand so the example is small enough to read. The math on top of them is genuine.