Context changes meaning
Compare “the key to the door” with “the key to the problem”. The same word appears in different surroundings. A useful representation of text needs to capture these relationships.
As you read a word, imagine other parts of the sentence lighting up around it. This is a useful analogy for attention, although model computations are not the same as human reading.
Inside a single step
In a simplified account, each token gets a query, key and value vector. Similarity between queries and keys helps determine the weights used to combine the values of available tokens.
An illustration of the idea, not measured attention weights from a particular model.
Multiple attention heads can build different combinations of features. In a causal language model, future tokens are hidden: generation can only use the available context.
What attention does not explain
Attention is one mechanism in a transformer, not a complete explanation of its behaviour. A single attention map does not reliably reveal why a model made a particular decision.