Retrieval-augmented generation (RAG)

01

When you ask an AI something it doesn't know

A company builds an internal assistant chatbot to answer employees' questions. Someone asks:

What's the hotel limit per night on a business trip to New York?

The company's travel policy says "Accommodation: up to $120 a night, except New York and San Francisco: $150." But the chatbot replies, "Up to $100 per night." Why does that happen?

Inside the chatbot is a large language model (LLM). An LLM has learned from a huge amount of public text, like web pages and books. But there are two things it can't know. The first is anything that happened after its training ended — it has only seen text up to that point. The second is anything that was never public in the first place, like this company's travel policy.

It would be nice if it just said "I don't know." But an LLM builds its answer by looking at the text so far and continuing with whatever is likely to come next. So even when it doesn't know, it produces a sentence that sounds plausible. An answer that states something false as if it were fact is called a hallucination.

The fix is simpler than you might think: send the relevant section of the policy along with the question. It's like an open-book exam. The model doesn't need to have memorized the policy. It just reads what's in front of it and answers.

The scene below shows this. On the left are the company's policy documents, the frame in the middle is the text sent to the model, and on the right are the AI model and its answer. At first, only the question is sent. Switch on sending the policy section too: the section moves into the text being sent, and a citation line runs from the answer back to that section.

This approach — retrieve the right material, use it to augment the text you send, then generate an answer from it — is called retrieval-augmented generation, or RAG for short. The name comes from a research paper published in 2020.

"Find the relevant section and attach it" actually hides several steps. Broadly, there's work you do ahead of time, before any question arrives, and work you do when a question comes in.

Why is the first step "split"? Couldn't you just attach the whole policy handbook? Not if it's hundreds of pages long.

02

Splitting documents into pieces (chunking)

There's a limit to how much text a model can read at once, called the context window. Today's models have fairly roomy context windows, but the longer the text you send, the slower the answer and the higher the cost. That's because you're usually billed by the amount of text (tokens) going in and out.

Sending more doesn't mean the model reads it all well, either. In a 2023 study by Liu et al., models made good use of information at the start and end of a long text but often missed information buried in the middle. Hunting for one line in hundreds of pages is exactly that situation.

So you split the documents into small pieces ahead of time, and when a question arrives you send only the few pieces that matter. Each piece is called a chunk, and splitting documents this way is called chunking.

The tricky part is the size. In the scene below, change the chunk size. For each size, it shows what happens when the one chunk closest to the question is attached.

If you split by single sentences, "Accommodation: up to $120 a night" and "Except New York and San Francisco: $150" end up in different chunks. Search brings back only the first one, which mentions accommodation, and the model says $120. Go the other way and make the whole chapter one chunk, and the part you need is in there — but so are the meals and transport sections. The text gets long and expensive, and other amounts easily slip into the answer.

In this example, one section per chunk worked best, because the rule and its exception sit in the same section. The right size depends on the document, but the rule of thumb is the same: make chunks big enough to hold everything needed to answer one question, and small enough to leave out what doesn't matter.

Ways to split

• Fixed-size: cut every so many characters or tokens (the units a model reads text in). It's the simplest, but it can cut a sentence in half.
• By document structure: cut along boundaries the document already has, like headings, sections and paragraphs. This suits well-structured documents such as policies and manuals.
• Where the meaning changes: find the points where the meaning shifts sharply from one sentence to the next and cut there. This is called semantic chunking.

Whichever way you split, you can repeat the last few sentences of one chunk at the start of the next, so nothing that sits on a boundary gets lost. This is called overlap.

The chunks are ready. Now, when a question comes in, you have to pick which of thousands of chunks to attach. But the employee asked about the "hotel", and the policy says "accommodation". A search that looks for the same words won't find this section.

03

Turning meaning into coordinates (embeddings)

Think of a map. Turn an address into two numbers, latitude and longitude, and you can measure how close two places are. You can do the same with text: turn its meaning into a set of numbers and use them like coordinates.

Turning text into a list of numbers like this is called embedding, and the resulting list of numbers is called a vector. The work is done by an embedding model, which is trained so that texts with similar meanings get similar numbers. So even though "hotel" and "accommodation" are different words, their meanings are close, and so are their coordinates.

The scene below shows the policy chunks embedded and plotted in space. Color shows the topic. Travel chunks cluster with travel chunks, leave chunks with leave chunks. Nobody told the model the topics; they gathered on their own because their meanings are similar. The lime point is the question. Try changing it.

Real embeddings aren't two or three numbers but hundreds to thousands. Nobody can draw a space like that, so the scene squeezes it down to three dimensions. The more numbers there are, the finer the shades of meaning they can capture.

The chunk embeddings are all calculated and stored before any question arrives. A store that holds vectors and lets you search them quickly is called a vector database (or vector store). When a question comes in, only the question is embedded on the spot.

Here's another one that trips people up on exams: an embedding model doesn't generate text. It only turns text into numbers so texts can be compared. Writing the answer is the job of the LLM in the next step.

Every chunk and the question are now points in the same space. All that's left is to find the chunk points closest to the question point.