Splitting data honestly

Your held-out score is only meaningful if the model genuinely knew nothing about that data. There are three routine ways that quietly stops being true — and every one of them makes your score better, which is why they survive review.

1. Random splits on time-ordered data

If you will predict the future, your test set must be the future. A random split puts January and March in training and February in test, so the model has seen "after" while predicting "before".

That is not a subtle advantage. Real deployment never offers it.

Split by time whenever the task is a prediction about later. Train on everything before a date, test after it. Your score will drop. The lower number is the true one, and finding it now is much cheaper than finding it in production.

2. Leakage through a feature

A feature that would not exist at prediction time is leakage.

The examples are always faintly embarrassing in hindsight:

• account_closed_date predicting churn.
• discount_applied, only ever set after a purchase, predicting purchase.
• Any column derived from the target, however indirectly.

The tell is a score that is too good. 0.99 on a genuinely hard problem is not a breakthrough; it is a bug, and it is nearly always this one.

Treat suspiciously good results as failures until you have explained them. The discipline here is entirely about resisting the wish to believe.

3. Leakage through preprocessing

Scale your features using the mean of the whole dataset, and the training data now carries information about the test set. Same for imputation, encoding categories, and feature selection.

The rule: fit every transformation on training data only, then apply it to test. A pipeline object that bundles the transformations with the model exists precisely so you cannot get this wrong by hand, and that is the reason to use one.

Groups that must not be split

If the same customer appears in many rows, a random split puts some of their rows in training and some in test. The model can learn that customer rather than the pattern, and it will be graded on people it has met.

Split by group — all of a customer's rows on one side. Same for a patient, a device, a shop.

The one that survives everything

Keep a final set you look at once, at the end.

Every time you tune against a validation set, you leak a little of it into your choices. Twenty rounds of "try it and see" is a slow fit to that set, and its score stops being honest well before you notice.

The final set is the only defence, and it only works if you are disciplined about "once". Looking twice and choosing the better run is exactly the thing it exists to prevent.

Try this

Take a model you already have and re-split it by time and by group. If the score holds, you have earned confidence you did not have. If it falls, you have learned something considerably more valuable than the old number.