Highlighted: embedding space where geometric relationships emerge
This is one of the most surprising things about embeddings. Take the classic example: king - man + woman ≈ queen. What’s actually happening in the vector math?
Each token’s embedding is a point in 8,192-dimensional space. The direction from one point to another is itself a vector — you get it by subtraction. So king - man gives you a vector that represents the difference between “king” and “man.” That difference turns out to point in roughly the same direction as queen - woman. The direction encodes the concept “royalty” or “gendered royal title” — and it’s consistent across pairs.
Nobody programmed these directions. They emerged because the model saw, across billions of sentences, that “king” and “queen” appear in similar contexts with a consistent gender-based offset, the same way “man” and “woman” do. The training process discovered that organizing the embedding space this way — where consistent relationships map to consistent directions — was useful for prediction.
The relationships aren’t stored as an explicit list. There’s no lookup table that says “king relates to queen.” The relationships are implicit in the geometry of the embedding space — the relative positions of the vectors. The embedding table gets reorganized during training (through gradient updates to its weights) so that these geometric regularities emerge naturally.
Other directions encode other relationships:
- Country → capital (France - Paris ≈ Germany - Berlin)
- Verb tense (walk - walked ≈ swim - swam)
- Comparative forms (big - bigger ≈ small - smaller)
These are all approximate — it’s not perfect arithmetic. But the fact that it works at all tells you something profound: the model learned a structured representation of meaning, not just a bag of associations.