Bit by Bit
← All courses
Machine Learning · Intermediate

Evaluating Models

A model is only as good as the way it is tested. Split data honestly, see why 99% accuracy can be useless, read a confusion matrix, and choose the threshold that fits the real cost of each mistake.

4 lessons28 minutesFree
Start this course →

Lessons

  1. 1Splitting the data honestlyUse training, validation and test sets without fooling yourself.7 min
  2. 2The accuracy trapSee why accuracy misleads, and what a confusion matrix shows instead.7 min
  3. 3Precision, recall and the thresholdChoose a threshold based on the real cost of each kind of mistake.8 min
  4. 4Cross-validationGet a more reliable score from limited data.6 min

What you will learn

Terms explained in this course

validation set
Examples held back from training and used to compare models and settings, so the test set stays untouched until the very end.
hyperparameter
A setting chosen before training rather than learned, such as k in k-nearest neighbours or the learning rate.
data leakage
When information from the test data, or from the future, slips into training, making a model look better than it really is.
accuracy
The share of all predictions that are correct.
class imbalance
When one class is much rarer than the other, such as fraud among normal payments.
baseline
A very simple reference model, such as always predicting the most common class. A real model must beat it to be useful.
confusion matrix
A table counting each kind of right and wrong prediction: true positives, false positives, false negatives and true negatives.
true positive
An item correctly flagged as positive.
true negative
An item correctly left unflagged.
f1 score
A single number that balances precision and recall: 2 × precision × recall ÷ (precision + recall).
cross-validation
Splitting the data into several parts, training on all but one and testing on the remaining one, in turn, then averaging the scores.