You've probably used one by now: ChatGPT, Claude, Gemini – these are all large language models, and they've gone from obscure research projects to mainstream tools in a remarkably short time. But how do they actually work?
LLMs don't understand, they predict
The most important thing to understand about a large language model is that it doesn't "think" in any meaningful sense. It predicts. Given everything that came before in a conversation, a large language model calculates what word – or more precisely, what token – is most likely to come next.
A token is the basic unit an LLM works with. It's not always a whole word – it might be a word fragment, a punctuation mark, or a common short word. The sentence "I like coffee" might be three tokens. A more complex technical term might be split into four or five.
This sounds almost too simple to produce the results these models achieve. But when you train a system on hundreds of billions of tokens of human text – books, articles, code, conversations – and optimize it to predict the next token with increasing accuracy, something interesting emerges: the model develops a remarkably sophisticated internal representation of how language works, how ideas connect, and how humans communicate.
Why the context window matters
Every LLM has a context window – the maximum amount of information it can "keep in mind" at once, measured in tokens. Think of it as working memory. Everything within the context window is available to the model when generating a response. Everything outside it isn't.
This has practical consequences. A very long document that exceeds the context window can't be processed all at once. A conversation that goes on long enough will eventually push earlier messages out of the window, and the model will effectively "forget" them. The size of the context window is one of the most practically significant differences between models.
Why LLMs hallucinate
Hallucinations are probably the most discussed limitation of LLMs. People often describe it as the model "making things up" or "lying," which implies intent. What's actually happening is more mechanical.
The model is always doing the same thing: predicting the most plausible next token. When it doesn't have reliable information about something – because it wasn't in the training data, or because the question is ambiguous – it doesn't stop and say "I don't know." It keeps predicting, generating text that sounds plausible but may be factually wrong.
This is a structural feature of how these models work, not a bug that can simply be patched. It's why verifying LLM output matters, especially for anything factual.
What fine-tuning does
A base LLM trained on general text data is capable but unfocused. Fine-tuning is the process of taking that base model and training it further on a more specific dataset, such as a company's internal documents, a particular domain's terminology, and examples of a desired style or format.
The result is a model that behaves differently in a specific context without having to be retrained from scratch. Fine-tuned models are how businesses adapt general-purpose LLMs to their own needs.
What this means in practice
The next time an LLM gives you a confident answer that turns out to be wrong, or phrases something in an unexpectedly human way, you'll know why. It's predicting the most plausible string of words, but does it so well it's easy to forget it cannot "think".
For a full reference of AI terms used in this article and beyond, see the AI Dictionary →