Tokenization & Embeddings

Before a neural network can process language, every word must become a number. This happens in two steps: Tokenization and Embedding.

Tokenization splits text into subword units using algorithms like Byte-Pair Encoding (BPE). BPE starts with characters and iteratively merges the most frequent adjacent pairs into new tokens. This gives a fixed vocabulary (GPT-4 uses ~100,000 tokens) that can represent any text: common words get their own token ("the" → [1234]), rare words split into meaningful pieces ("unbelievable" → ["un", "believ", "able"]), and nothing is ever out-of-vocabulary. A typical English sentence requires roughly 1.3–1.5 tokens per word.

Each token ID is then looked up in an Embedding Table — a learned matrix of shape (vocab_size × embed_dim). The embedding dimension ranges from 512 (small models) to 12,288 (GPT-4). These dense vectors are learned during training so that tokens with similar meanings end up geometrically close. The famous example: king − man + woman ≈ queen in the vector space.

Embeddings capture semantic relationships that no integer encoding could: synonyms cluster together, antonyms are in predictable directions, grammatical forms follow consistent patterns. The embedding layer is often the single largest component of a language model.