Super Convergence with Cyclical Learning Rates in TensorFlow

Super Convergence with Cyclical Learning Rates in TensorFlow

Super-Convergence using Cyclical Learning Rate schedules is one of the most useful techniques in deep learning and very often overlook. Chances are you missed it as well. Then this article is for you.

Super-Convergence using Cyclical Learning Rate schedules is one of the most useful techniques in deep learning and very often overlooked. It allows for rapid prototyping of network architectures, loss function engineering, data augmentation experiments and training production ready models in orders of magnitude less training time and epochs.

It is demonstrated within this article and the accompanying code that it is possible to train deep neural network models for complex tasks such as super resolution from initialised weights (i.e. not pre-trained or transferred) within minutes.

The techniques for training in this way are from research that is a few years old. In 2015 Leslie Smith published a paper titled Cyclical Learning Rates for Training Neural Networks that outlines methods to train deep neural network models quicker (with fewer epochs) by cycling up to very high learning rates. Sadly this went widely unnoticed by the majority of deep learning practitioners.

This technique is so effective that a state of the art super resolution model can be trained in as low as 16 epochs in approximately 4 minutes from initialisation not using a pre-trained model. With impressive results such as the following.

artificial-intelligence deep-learning computer-vision machine-learning

What is Geek Coin

What is GeekCash, Geek Token

Best Visual Studio Code Themes of 2021

Bootstrap 5 Tutorial - Bootstrap 5 Crash Course for Beginners

Nest.JS Tutorial for Beginners

Hello Vue 3: A First Look at Vue 3 and the Composition API

How are deep learning, artificial intelligence and machine learning related

What is the difference between machine learning and artificial intelligence and deep learning? Supervised learning is best for classification and regressions Machine Learning models. You can read more about them in this article.

Machine Learning Vs Deep Learning: Difference Between Machine Learning and Deep Learning

This article will simply explain the concept which will help you understand the difference between Machine Learning and Deep Learning. 

Artificial Intelligence, Machine Learning, Deep Learning 

Artificial Intelligence (AI) will and is currently taking over an important role in our lives — not necessarily through intelligent robots.

Start a Career in Machine Learning and Artificial Intelligence

Enroll now at best Artificial Intelligence training in Noida, - the best Institute in India for Artificial Intelligence Online Training Course and Certification.

AI(Artificial Intelligence): The Business Benefits of Machine Learning

Enroll now at CETPA, the best Institute in India for Artificial Intelligence Online Training Course and Certification for students & working professionals & avail 50% instant discount.