Evaluating Models
A model is only as good as the way it is tested. Split data honestly, see why 99% accuracy can be useless, read a confusion matrix, and choose the threshold that fits the real cost of each mistake.
Start this course →Lessons
- 1Splitting the data honestlyUse training, validation and test sets without fooling yourself.7 min
- 2The accuracy trapSee why accuracy misleads, and what a confusion matrix shows instead.7 min
- 3Precision, recall and the thresholdChoose a threshold based on the real cost of each kind of mistake.8 min
- 4Cross-validationGet a more reliable score from limited data.6 min
What you will learn
- Train on the training set, tune on the validation set, report on the test set once.
- A hyperparameter is a setting you choose, so choosing it uses up validation data.
- Data leakage makes a model look better than it is.
- Accuracy can look excellent under class imbalance while the model is useless.
- Always compare with a baseline.
- A confusion matrix shows every kind of mistake separately.
- Raising the threshold usually raises precision and lowers recall.
- The F1 score balances the two.
- The right threshold depends on which mistake is more costly in the real world.
- Cross-validation trains and tests several times on different splits, then averages.
- It gives a steadier estimate than one split, especially with little data.
- Keep a final test set aside even when you use cross-validation.
Terms explained in this course
- validation set
- Examples held back from training and used to compare models and settings, so the test set stays untouched until the very end.
- hyperparameter
- A setting chosen before training rather than learned, such as k in k-nearest neighbours or the learning rate.
- data leakage
- When information from the test data, or from the future, slips into training, making a model look better than it really is.
- accuracy
- The share of all predictions that are correct.
- class imbalance
- When one class is much rarer than the other, such as fraud among normal payments.
- baseline
- A very simple reference model, such as always predicting the most common class. A real model must beat it to be useful.
- confusion matrix
- A table counting each kind of right and wrong prediction: true positives, false positives, false negatives and true negatives.
- true positive
- An item correctly flagged as positive.
- true negative
- An item correctly left unflagged.
- f1 score
- A single number that balances precision and recall: 2 × precision × recall ÷ (precision + recall).
- cross-validation
- Splitting the data into several parts, training on all but one and testing on the remaining one, in turn, then averaging the scores.