In this article, we will focus on adding and customizing Early Stopping in our machine learning model and look at an example of how we do this in practice with Keras and TensorFlow 2.0.

Introduction to Early Stopping

In machine learning, early stopping is one of the most widely used regularization techniques to combat the overfitting issue.

Early Stopping monitors the performance of the model for every epoch on a held-out validation set during the training, and terminate the training conditional on the validation performance.

From Hands-on ML [1]

Early Stopping is a very different way to regularize the machine learning model. The way it does is to stop training as soon as the validation error reaches a minimum. The figure below shows a model being trained.

As the epochs go by, the algorithm leans and its error on the training set naturally goes down, and so does its error on the validation set. However, after a while, the validation error stops decreasing and actually starts to go back up. This indicates that the model has started to overfit the training data. With Early Stopping, you just stop training as soon as the validation error reaches the minimum.

It is such a simple and efficient regularization technique that Geoffrey Hinton called it a “beautiful free lunch.” [1].

With Stochastic and Mini-batch Gradient Descent

With Stochastic and Mini-batch Gradient Descent, the curves are not so smooth, and it may be hard to know whether you have reached the minimum or not. One solution is to stop only after the validation error has been above the minimum for some time (when you are confident that the model will not do any better), then roll back the model parameters to the point where the validation error was at a minimum.

In the following article, we are going to add and customize Early Stopping in our machine learning model.

Environment setups and dataset preparation

We will be using the same dataset as we did in the model regularization and batch normalization. You can skip this chapter if you are already familiar with it.

In order to run this tutorial, you need to install

TensorFlow 2, numpy, pandas, sklean, matplotlib

They can all be installed directly vis PyPI and I strongly recommend to create a new Virtual Environment. For a tutorial on creating a Python virtual environment

• Create Virtual Environment using “virtualenv” and add it to Jupyter Notebook
• Create Virtual Environment using “conda” and add it to Jupyter Notebook

#early-stopping #keras #tensorflow #machine-learning #data-science