Highlighted: every step that uses learned weight matrices
A weight is a single number that the model can adjust to change its behavior. That’s it. When people say a model has “405 billion parameters,” they mean it has 405 billion individual numbers — weights — that collectively define everything the model does.
Weights live in matrices (grids of numbers). The embedding table you just learned about is one matrix of weights — 128,000 rows × 8,192 columns = ~1 billion weights, just for embeddings. But that’s a tiny fraction. The vast majority of weights live in the model’s layers (more on those in model layers), where they define the transformations that happen to your token vectors as they flow through the network.
Here’s the important mental model: the weights are the model. The code that runs an LLM is relatively simple — it’s matrix multiplications and a few other operations in a loop. What makes GPT-4 different from a random noise generator is the values of the weights. Two models with identical code but different weights will produce completely different outputs. Training is the process of finding weight values that produce useful behavior. The architecture (code) defines what operations happen; the weights define what those operations do.
Every weight started as a small random number. Through training, each one got nudged — millions of times — toward a value that makes the model slightly better at predicting the next token. The final trained weights encode everything the model “knows.”