Transformers

The Transformer architecture ("Attention Is All You Need", 2017) replaces recurrence entirely with attention. Its core innovation is Multi-Head Attention: instead of one attention operation, the model runs several in parallel, each with its own learned Q, K, V projections. The outputs of all heads are concatenated and projected — giving the model multiple "perspectives" on the same sequence simultaneously.

Each Transformer block contains two sub-layers: Multi-Head Attention followed by a Feed-Forward Network (FFN) — two linear layers with a non-linearity (usually GELU) between them. Critically, both sub-layers use residual connections (y = F(x) + x) and Layer Normalization, which stabilize training in very deep networks and allow gradients to flow directly.

Since Transformers process all tokens simultaneously (unlike RNNs), they have no inherent sense of order. Positional Encoding is added to each token's embedding to inject position information. The original paper used sinusoidal functions; modern models often use learned embeddings or rotary position encodings (RoPE).

The architecture splits into Encoders (BERT — bidirectional, good for understanding) and Decoders (GPT — autoregressive, good for generation). Modern LLMs like GPT-4 stack 96+ decoder blocks, each with 96 attention heads.