Training, Testing and Overfitting

Introduction

How do you know a model is really good? If you test a student with the exact questions they practised, a high score proves little. Machine learning has the same problem and the same fix.

The simple explanation

Train the model on one part of the data, then test it on a part it has never seen.

• Training data is what the model learns from. Often about 70 to 80 percent.
• Test data is kept hidden until the end, then used once to see how well the model copes with new examples.

If a model does well on test data, it has probably learned a real pattern.

Overfitting

Overfitting happens when a model memorises the training examples, including their noise and quirks, instead of learning the general pattern. It scores very well in training and badly on new data. It is like a student who memorised the practice answers but did not understand the topic.

Underfitting is the opposite: the model is too simple and misses the pattern even in the training data.

Ways to reduce overfitting include getting more varied data, using a simpler model, and checking the score on data held back from training.

Measuring a model

The right measure depends on the job:

• Accuracy: the share of predictions that were right. Simple, but it can mislead. If only 1 percent of emails are spam, a model that never flags spam is 99 percent accurate and useless.
• Precision and recall: precision asks "of the things I flagged, how many were truly spam?". Recall asks "of all the spam, how much did I catch?".

Try it yourself

You have 100 labelled photos. Say how you would split them for training and testing, and why you must not peek at the test photos while building the model.

Summary

• Split data into training and test sets, and test only on unseen data.
• Overfitting means memorising instead of learning the pattern.
• Choose a measure, such as precision and recall, that fits the real problem.