How to effectively and efficiently use data generators in Keras for Computer Vision applications of Deep Learning

I have worked as an academic researcher and am currently working as a research engineer in the Industry. What my experience in both of these roles has taught me so far is that one cannot overemphasize the importance of data generators for training. Right from the MNIST dataset which has just 60k training images to the ImageNet dataset with over 14 million images a data generator would be an invaluable tool for deep learning training as well as inference. Few of the key advantages of using data generators are as follows:

• Allows the use of multi-processing: Parallelize the loading process across multiple CPU cores to make the process faster.
• Allows you to generate batches: You can use smaller chunks of data to train your model using batch gradient descent. And practically all real world datasets pertaining to deep learning generally can’t fit into memory in one go, so this is the only possible solution most times.
• Allows you to do data augmentation: Regardless of whether you have limited data or whether you want to add variety in terms of noise augmentation is what you would normally do. And data generators help to do these both statically as well as on the fly by using the multi-processing capabilities of the modern CPUs.
• No need to write boilerplate code: Most deep learning frameworks support their version of data loaders/data generators and this reduces the time you would have to write boilerplate code to handle the create the creation of batches of data and its augmentation in a parallelized manner. This also introduces consistency and readability in your code.

In this article, I discuss how to use DataGenerators in Keras for image processing related applications and share the techniques that I used during my researcher days.

#data-science #deep-learning #machine-learning #artificial-intelligence #keras