Grounding and Retrieval

Introduction

A model's built-in knowledge is general, can be outdated and knows nothing about your documents. If you ask about your company's policy, it may invent a plausible answer. The fix is to ground the model: give it the relevant source text and tell it to answer from that.

The simple explanation

Grounding means answering from supplied evidence, instead of from the model's memory.

The simplest version needs no special tools. Paste the document into the prompt:

Answer the question using ONLY the text between the triple quotes.
If the answer is not there, say "Not in the provided text."
Quote the sentence you used.

"""
[policy text here]
"""

Question: How many days of leave do new staff get?

The quote requirement makes the answer easy to verify.

When the documents are too large

You cannot paste a whole library into the context window. Retrieval-augmented generation (RAG) solves this by finding only the relevant pieces first.

The steps:

1. Split your documents into chunks, such as paragraphs.
2. Convert each chunk into an embedding, a list of numbers that captures its meaning.
3. Store the embeddings in a searchable index, often called a vector database.
4. When a question arrives, convert it into an embedding, and find the closest chunks by meaning.
5. Put those chunks in the prompt with the question, and ask the model to answer from them and cite them.

Embeddings and semantic search

An embedding places text in a space where similar meanings are close together. "How do I reset my password?" and "I forgot my login" are different words with similar meaning, so they land near each other. This is semantic search: it finds by meaning, not by exact keywords.

Good systems often combine semantic search with ordinary keyword search, because names, codes and exact phrases still matter.

Making RAG work well

• Chunk size matters. Too small loses context, and too big wastes the window.
• Keep documents current and remove outdated ones.
• Show sources to users, so they can check.
• Let the system say "I don't know" when the retrieved text does not answer the question.
• Check retrieval separately from generation. If the right chunk is not found, the best model cannot answer correctly.
• Respect permissions. Only retrieve documents the asking user is allowed to see.

Limits

Grounding reduces errors, but does not remove them. The model can still misread a passage, mix two sources or ignore the instruction. For important uses, test with real questions and keep a human in the loop.

Other ways to ground

• Web search tools, so the model can look up current information
• Database or API calls for live data such as prices or order status
• Calculators and code execution for exact numbers

Try it yourself

Take a one-page document you know well. Ask an AI three questions about it using the grounding prompt above. Check each answer against the text, and add one question whose answer is not in the document, to see whether it says "not in the provided text".

Summary

• Grounding makes the model answer from supplied evidence, and asking for quotes makes checking easy.
• RAG retrieves relevant chunks using embeddings and semantic search, then answers from them.
• Test retrieval and answers separately, show sources and respect access permissions.