Transformers and multi-head attention

The attention topic showed a single attention pattern: each word looking at the other words and deciding which ones matter. A transformer does not run just one of these. It runs many attention patterns at the same time, called heads, and each head learns to watch a different kind of relationship.

Many heads, one sentence

A single head produces one set of links over the sentence. With several heads, the same sentence gets several overlays at once. One head might track coreference, linking a pronoun to the noun it stands for, so "it" reaches back to "animal". Another might track position, with each word mostly looking at the word just before it. Another might track grammar, linking a verb to its subject and object. Each head sees the same words but draws a different map.

This matters because language has many kinds of structure at the same time. No single attention pattern can capture coreference, word order, and grammar together, so the model splits the work across heads and lets each specialize.

Heads run in parallel, then layers stack

Within one layer, the heads run in parallel and their results are combined, so the model can use several relationships for the same word at once. Then transformers stack many such layers, and later layers attend over the outputs of earlier ones. Early layers tend to capture local and surface patterns, while later layers capture longer-range and more abstract structure built from what came before.

The attention topic is one head in one layer. A transformer is many heads in parallel, repeated across many layers. That combination, simple attention multiplied out and stacked, is what lets these models handle the structure of real language.

A single attention pattern can only capture one kind of relationship. Real transformers use several attention heads at the same time, each learning to look for something different. This is called multi-head attention, and the heads all run in parallel over the same sentence.

The figure shows three heads side by side, each looking at the same sentence "the animal didn't cross the street because it was tired." Click any word and watch how each head connects it differently.

The first head tracks coreference, which word a pronoun refers to. Click "it" and this head points back to "animal," because that is what "it" means here. The second head tracks the previous word: each word simply looks back at the one before it. The third head tracks verbs and their arguments. Click "cross" and it points to "animal," the thing doing the crossing.

Same sentence, same word, three different answers. That is the power of multiple heads. One head follows grammar, another follows meaning, another just tracks position, and the model gets all of these views at once instead of being forced to pick one. When the heads are combined, the word carries all of these relationships together.

The three heads here are hand-designed to show three clear, recognizable jobs: coreference, previous-word, and verb-to-argument. In a real transformer the heads are not assigned jobs; each one learns its own pattern from data, and the patterns are often messier and harder to name than these. The links you see are computed from simple rules chosen to make each head's job legible. The idea they illustrate is real: many heads, running in parallel over the same words, each capturing a different kind of relationship.