Tokens, Context and Sampling

Introduction

Knowing a little about what happens inside a language model explains many behaviours that seem odd: why long chats drift, why answers vary, why it is bad at counting letters. You do not need the maths, only the key ideas.

Tokens

Models do not read letters or whole words. They read tokens, chunks of text. Common words are often one token, and rare words are split into several. As a rough guide for English, a token is about three quarters of a word.

• Costs and limits are counted in tokens.
• Other languages and code may use more tokens per word.
• Because text is split into chunks, tasks about individual letters, such as counting the letters in a word, are surprisingly hard for models.

The context window

The context window is the maximum amount of text, in tokens, the model can consider at once. It includes everything: your instructions, earlier messages, documents you pasted and the answer it is writing.

• If a conversation exceeds the window, older parts are dropped or summarised, and the model "forgets" them.
• Even inside the window, models can pay less attention to material in the middle of a very long input.
• Put the most important instructions clearly, and repeat key constraints if the conversation is long.
• Starting a fresh chat for a new topic often gives better results.

How the next token is chosen

A model repeatedly predicts a probability for each possible next token, then picks one. How it picks is controlled by sampling settings:

• Temperature controls randomness. Low (near 0) makes the most likely choice almost every time, so answers are focused and repeatable. High makes less likely tokens more common, so output is more varied and creative, and also more error-prone.
• Top-p (nucleus sampling) limits the choice to the smallest set of tokens whose probabilities add up to p.
• Max tokens caps the length of the answer.

Use low temperature for facts, extraction and code. Use higher temperature for brainstorming and creative writing. Even at temperature zero, outputs may not be identical every time.

How models are made

1. Pre-training: the model learns to predict the next token over a huge amount of text. This gives it language, facts and patterns, and it is very costly.
2. Fine-tuning: further training on examples of the behaviour wanted, such as following instructions and answering helpfully.
3. Preference tuning: humans, or other models, compare answers, and the model is adjusted toward the preferred ones. This is part of how assistants become polite and safe to use.
4. Safety work and evaluation: testing for harmful, biased or false output, and adding guardrails.

The model has a knowledge cutoff: it knows nothing after its training data ends, unless it is given tools or documents. It does not look things up unless it is connected to search.

Why hallucinations happen

The model is trained to produce plausible text, not to check truth. When a fact is rare, missing or ambiguous, the most plausible-sounding continuation can still be wrong. Reducing this means giving it the source material, which is the next lesson.

The transformer in one paragraph

Most modern models use the transformer design from 2017. Its key idea, attention, lets each token look at every other token in the context and weigh how relevant they are. That is how the model can connect a pronoun to the right noun many words earlier.

Try it yourself

Ask the same factual question three times with a high temperature setting if your tool allows it, and three times with a low one. Compare how much the answers differ, and note which kind of task each setting suits.

Summary

• Models read tokens, and the context window limits how much they can consider.
• Temperature and top-p control randomness, so use low values for accuracy.
• Models are pre-trained, fine-tuned and preference-tuned, and they predict plausible text, which is why they can be wrong.