Why and when training, validation, and testing splits are needed and how to build them from a tf.data.Dataset using Python

Why and when do we need train, validation, and test splits?

One of the biggest challenges when developing a machine learning model is to prevent it from overfitting to the data set. The difficulty arises when the model learns a combination of weights that performs well on the data used for training but fails to generalize when the model is given images it has never seen. This is known as overfitting.

When implementing a model that will be deployed in the real world, we might want to have an estimate of how it will behave once it is put into production. This is where the test set comes into play, a random partition of the original dataset that is intended to represent data not used for training, so that we can have an estimate of how our model will behave with unseen data.

In addition, there is a third set that is useful when we plan to experiment with different configurations of our model, such as alternative architectures, optimizers, or loss functions, also known as hyperparameter-tuning. To compare the performance of these experiments, another random split can be extracted from the original data set, which is not used for training nor testing but to validate our model in different configurations. This is known as the validation set.

Now, you might be wondering, but then, validation and test sets have the same purpose, right? Well, it is true that both datasets serve to have an estimation of how our model performs on data that have not been used for training. However, when trying different model configurations to have the best validation metrics, we are in a way fitting our model to the validation set, choosing the combination of parameters with the best performance on that set.

Once we have run our hyperparameter-tuning and have the model that performs best, the test set allows us to get an idea of how well this model will perform in production. Therefore, it should only be used at the end of the project.

#metrics #training #machine-learning #data-science #tensorflow