Optimization Algorithms for machine learning are often used as a black box. We will study some popular algorithms and try to understand the circumstances under which they perform the best.

The purpose of this blog is to:

• Understand Gradient descent and its variants, such as _Momentum, Adagrad, RMSProp, NAG, _and Adam;
• Introduce techniques to improve the performance of Gradient descent; and
• Summarize the advantages and disadvantages of various optimization techniques.

Hopefully, by the end of this read, you will find yourself equipped with In with intuitions towards the behavior of different algorithms, and when to use them.

#optimization #deep-learning #rmsprop #adam #gradient-descent