Gradient Descent- A Race to find the Global Minimum

Gradient Descent- A Race to find the Global Minimum

A basic neural network basically comprises of the input layer, the hidden layer(s) and finally the output layer with some network parameters (weights and biases).

We can see the presence of a loss function i.e the difference between the actual value(y) and the predicted value(ŷ) from the neural network.

In order to reduce this loss we use an optimizer.There are various types of optimizers one of them being Gradient Descent.

Gradient Descent is a very generic optimization algorithm capable of finding optimal solutions to a wide range of problems.

The general idea of Gradient Descent is to tweak parameters iteratively in order to minimize the loss function.

Image for post

In short what the Gradient descent does is to find out the best weights(or parameters) in order to reduce the loss function by modifying all the weights(or parameters) with the help of backpropagation through the network.

Image for post

Eq1: Modifying old weights to obtain new weights

Here λ is known as the learning rate.

How does Gradient Descent help us?

Suppose you are lost in the mountains in a dense fog, you can only feel the slope of the ground below your feet. A good strategy to get to the bottom of the valley quickly is to go downhill in the direction of the steepest slope. This is exactly what Gradient Descent does: it measures the local gradient of the error function with regards to the parameter θ, and it goes in the direction of descending gradient.Once the gradient is zero, you have reached a minimum.

gradient-descent regression machine-learning normalization deep-learning deep learning

Bootstrap 5 Complete Course with Examples

Bootstrap 5 Tutorial - Bootstrap 5 Crash Course for Beginners

Nest.JS Tutorial for Beginners

Hello Vue 3: A First Look at Vue 3 and the Composition API

Building a simple Applications with Vue 3

Deno Crash Course: Explore Deno and Create a full REST API with Deno

How to Build a Real-time Chat App with Deno and WebSockets

Convert HTML to Markdown Online

HTML entity encoder decoder Online

ML Optimization pt.1 - Gradient Descent with Python

In this article, we explore gradient descent - the grandfather of all optimization techniques and it’s variations. We implement them from scratch with Python.

All about Gradient Descent in Machine Learning and Deep Learning!

Ever wondered how the machine learning algorithms give us the optimal result, whether it is prediction, classification or any other? How…

Hire Machine Learning Developers in India

We supply you with world class machine learning experts / ML Developers with years of domain experience who can add more value to your business.

Applications of machine learning in different industry domains

We supply you with world class machine learning experts / ML Developers with years of domain experience who can add more value to your business.

Hire Machine Learning Developer | Hire ML Experts in India

We supply you with world class machine learning experts / ML Developers with years of domain experience who can add more value to your business.