Fitting, and the only thing that matters

A model takes inputs and produces an output. Training adjusts it so its outputs match the examples you gave it.

Here is the part that is not obvious: matching the training examples is not the goal, and getting better at it eventually makes the model worse.

The demonstration

Take ten points that follow a straight line, with a little noise.

Fit a straight line: it misses each point slightly, because of the noise.

Fit a ninth-degree polynomial: it passes through all ten exactly. Zero error. By the only measure you have looked at so far, it is a perfect model.

Now ask both for a prediction at a new x, slightly outside your points. The line gives something sensible. The polynomial shoots off to an absurd value.

The polynomial did not learn the pattern. It learned the noise — memorised your ten points, including the random wobble that will never repeat. That is overfitting, and it is not a corner case; it is the default outcome of optimising hard enough on the data in front of you.

Signal and noise

Every dataset is signal — the real relationship — plus noise, which is measurement error, randomness, and things you did not record.

A model with enough capacity will fit both, and it cannot tell them apart, because nothing in the training process distinguishes them. Your job is to give it enough capacity for the signal and not so much that it consumes the noise.

Which is why you hold data back

Split the data. Train on one part, evaluate on a part the model has never seen.

The held-out error is the only number that means anything, because it is the only one that answers the actual question: how will this behave on data it has not seen? That is the question every deployed model is asked, every time.

Training error tells you how well it memorised. That is useful only as a diagnosis when compared against held-out error:

• Training error high, held-out error high → underfit. Not enough capacity, or the features do not carry the signal.
• Training error low, held-out error high → overfit. Classic.
• Both low → good, if the split was honest.

That last condition is doing enormous work, and it is where the next lesson goes.

The baseline you must beat

Before any model: what does the trivial answer score?

Always predict the mean. Always predict the most common class. Repeat yesterday's value. Run it and write the number down.

An alarming number of production models do not beat that, and nobody checked. It costs five minutes and it occasionally saves the entire project.

Try this

Take any dataset, fit the simplest possible model and a much more complex one, and compare both on held-out data. Watching the complex one lose is worth more than reading about it.