Attention Mechanism

The Attention Mechanism was the breakthrough that enabled modern language models. Instead of compressing an entire sequence into one hidden state (as RNNs must), attention allows a model to directly reference any position in the sequence when producing each output — with a learned, input-dependent weighting.

Each token produces three learned projections: a Query (Q) ("what am I looking for?"), a Key (K) ("what do I contain?"), and a Value (V) ("what is my actual content?"). To compute attention from token A to token B, we take the dot product of A's Query with B's Key to get a score.

The full formula is: Attention(Q, K, V) = softmax(QK^T / √d_k) · V. We divide by √d_k (the key dimension) to prevent dot products from growing so large that the softmax gradient vanishes. The softmax turns raw scores into a probability distribution — the attention weights — which sum to exactly 1. These weights are used to compute a weighted sum of the Values: the final context vector.

Self-attention is when Q, K, V all come from the same sequence — each position attends to all others. This enables the model to build a rich, context-aware representation of every token.