What the model is actually doing
Next-token prediction, why that makes it fluent and unreliable in the same breath, and what follows for anything you build.
Claude produces text one token at a time, each one conditioned on everything before it. That is the whole mechanism. Everything useful and everything dangerous about it falls out of that sentence.
Why it is so good
Predicting the next token well, across enormous amounts of text, requires representing a great deal: grammar, structure, argument, code semantics, the shape of a good explanation. You cannot continue a proof or a function convincingly without something that behaves like understanding of it.
So the capability is real. This is not a party trick, and dismissing it as "autocomplete" will make you build worse things.
Why it is confidently wrong
Nothing in that process checks a claim against the world. The model produces the continuation that best fits the pattern, and a fluent false statement fits the pattern of confident prose exactly as well as a true one.
Crucially, it has no signal for its own uncertainty that is reliable enough to build on. It is not lying — lying requires knowing. It is producing plausible text, and plausible is not the same as correct.
The three failure modes you will actually hit
- Fabricated specifics. Citations, function names, API parameters, statistics. The more precise-looking the detail, the more it is worth checking — precision is a style the model can produce without having the fact.
- Stale knowledge. It knows the world up to its training cut-off and does not know that it does not know what happened after.
- Agreeable drift. Push back on a correct answer and it may well fold. That is not it deferring to superior evidence; it is the pattern of a conversation where one party is corrected.
What this means for what you build
The design principle for the whole course:
Give the model the facts, and use it for language and judgement — not for recall.
Almost every reliable application of an LLM is a version of that. Retrieval puts your documents in front of it. Tool use lets it call something that knows. Structured output constrains what it can return. Evaluation catches drift.
The applications that fail are the ones that asked the model to remember something and then trusted the answer.
And where a human stays
If a wrong output causes real harm — money moving, a medical or legal statement, an irreversible action — a person approves it. Not because the model is bad, but because it has no way to tell you which of its outputs is the wrong one.