Underneath everything a large language model does is one deceptively small task: predict the next word.
That’s it. That’s the training objective. Given “the cat sat on the”, assign probabilities to what comes next: “mat” high, “roof” lower, “carburettor” very low. The startling discovery of the last few years is how much apparent understanding falls out of doing that one task extremely well at extreme scale.
My old whiteboard sketch of the objective survives, and it still explains it faster than a paragraph:
The old way, and my own small museum piece
Language modelling long predates the current excitement. The classical approach was statistical: n-gram models, which estimate the next word from counts of the previous few. Simple, fast, and hopeless at anything requiring long-range memory. By the time an n-gram model finishes a sentence, it has forgotten how the sentence began.
I have a personal exhibit here. Over a decade ago I built a small word-prediction model in R and put it on the web, and it’s still running. Type a phrase and it guesses your next word from n-gram statistics. Playing with it after using a modern model is a useful kind of time travel: you can feel exactly what scale and architecture bought us.
What changed
Neural language models replaced counting with learned representations: words became vectors, and the model learned relationships between them. Then transformers arrived and removed the memory bottleneck, letting the model attend to everything in the context at once. Three names from this era are worth knowing: GPT-3, which showed that 175 billion parameters of pure next-word prediction could write, translate and answer questions it was never explicitly taught; BERT, Google’s model that reads in both directions and quietly improved search; and ELMo before both, which at about 94 million parameters proved contextual embeddings mattered. The parameter counts are the “knobs” the training process tunes — more of them means more capacity, more cost, and more data required to fill them.
How do you measure such a thing? The standard metric is perplexity, which is roughly how surprised the model is by text it hasn’t seen. Lower is better. It’s a narrow ruler for what these systems became, and the field has been arguing about better ones ever since.
Where they were already showing up
Even in 2022 the applications were everywhere once you knew to look: speech recognition, machine translation, autocomplete, chatbots, the ranking layer inside search engines. The pattern was consistent. Anywhere language needed turning into probabilities, a language model was doing the turning.
What I’d tell anyone trying to build intuition: don’t start from the demos, start from the objective. A model that predicts next words has learned, as a side effect, an enormous amount about how the world tends to be described. That’s genuinely powerful. It’s also exactly why these systems state falsehoods with perfect fluency: plausible continuation was the goal, and truth only correlates with it. Keep that one sentence and most of what these models do, and do wrong, stops being mysterious.