Tokens, Context and Hallucination: How LLMs Really Work

You do not need a maths degree to use LLMs well, but a working mental model prevents most mistakes.

Prediction, not retrieval

An LLM does not look up answers. It predicts the next token — roughly a word fragment — given everything before it. Fluency and accuracy are separate properties, which is exactly why a confident wrong answer is possible.

The context window

Everything the model can "see" at once lives in its context window: your prompt, attached documents, and the conversation so far. Anything outside it does not exist to the model. Long documents therefore need chunking and retrieval rather than being pasted wholesale.

Grounding beats prompting

Give the model sources instead of asking it to remember.

Ask for citations so claims can be checked.

Require a second pass where the model verifies its own output against the source.

Where hallucination comes from

Hallucination is what happens when the model has no grounded information and still has to produce the next token. Reduce it by narrowing the question, supplying data, and building a checking step into the workflow — which is precisely what an agent does and a raw chat window does not.