How large language models work

01

An LLM picks what comes next

LLM stands for large language model. "Large" refers to two things: the amount of text it was trained on, and the number of numbers inside the model. Those numbers are called weights, and we'll come back to them in chapter 5.

An LLM really only does one thing: it looks at the text so far and picks the one token that comes next. A token is the unit a model reads and writes in. For now, think of it as a piece of a word. It's a lot like your phone keyboard suggesting the next word, except the model looks at much longer text and chooses far more carefully.

In the scene below, the tiles on the left are the text we typed: "The weather is". The bars behind them are candidates for the next slot, and each bar's height is the probability the model gave it. Step through it one token at a time.

Once a token is picked, it's added to the end of the text, and the model looks at the whole, longer text again to pick the next one. This repeats until the answer is done. Feeding each output back in as the next input is called autoregressive generation. It's also why chatbot answers trickle onto the screen bit by bit instead of appearing all at once: they really are being produced one token at a time.

So when does it stop? The model also has a special token that means "the end". It stops when that token gets picked, as in the last step of the scene, or when it hits a length limit you set.

A large model that's pre-trained on a huge amount of data and then reused for many different tasks, like summarizing, translating or answering questions, is called a foundation model. An LLM is a foundation model for text. Models that also handle images or audio are called multimodal models.

In the scene, tokens looked like whole words. That was a simplification to make things easier to follow. Real tokens look a little different.

02

Cutting text into tokens

A model can't read letters as they are. The text is first cut into small pieces, and each piece is swapped for a fixed number. Those pieces are tokens, and the tool that does the cutting is called a tokenizer. So the model doesn't receive text at all — it receives a list of numbers.

A token can be a whole word, part of a word, or a punctuation mark. Common chunks of letters become a single token, while rare words get split into several pieces. That way, even a word the model has never seen can be built from pieces it knows. In the scene below, switch between sentences and watch them get cut up. The number under each tile is the token ID the model actually receives (the IDs and the places where words are split are examples; they differ from model to model).

"Unbelievable" is one word, but it became three tokens. The Korean sentence, which means the same thing, has about half as many characters, yet it needs more tokens. Many tokenizers were built from mostly English text, so other languages often use more tokens to say the same thing. Numbers aren't split digit by digit either; they're cut up irregularly. That's one reason LLMs struggle with big arithmetic or counting letters.

Three things the token count decides

Now the model receives a list of numbers. But those numbers are just labels. Looking at 3022 and 3023, you can't tell whether their meanings are close. We need a step that puts meaning into the numbers.

03

Numbers that carry meaning (embeddings)

Inside the model there's one big table. It has a row for every token ID, and each row holds hundreds to thousands of numbers. When a token ID comes in, the model pulls out that row. Turning a token into a list of numbers like this is called embedding, and the list of numbers you get is called a vector.

At first, the numbers in the table are random. During training (chapter 5), they're nudged little by little so that tokens that often show up in similar text end up with similar-looking rows. The scene below pulls two words' rows out of the table and draws each number as a bar. Real rows have far more numbers; here they're cut down to 10.

"Cat" and "dog" have almost the same bar pattern. "Cat" and "car" look quite different. How alike two vectors are is usually measured with cosine similarity: the closer to 1, the more alike. Here's the bit that trips people up: individual slots don't have labels like "animal slot" or "vehicle slot". The meaning comes from many slots working together.

Embeddings show up in retrieval-augmented generation (RAG) too. There, a dedicated embedding model turns a whole sentence or paragraph into a single vector to find documents whose meaning is close to the question. Same principle: close in meaning, close in numbers. There's more in the embeddings chapter of the RAG guide.

The same word "spring" means different things depending on the words around it. How does the model notice the difference?