Understanding Gradients in Machine Learning

Understanding Gradients in Machine Learning

Understanding Gradients in Machine Learning. Taking derivatives of tensor-valued functions, with examples.

When I first began studying neural networks, I was immediately confronted with formulas for backpropagating gradients, starting from the loss function computed at the end of the network and working back layer-by-layer. These formulas already looked quite complex even despite making several basic assumptions — such as a fully-connected network and sigmoid activation functions after each neuron. I wondered how popular packages like TensorFlow and Pytorch perform the same operations but for arbitrary mathematical functions.

Modern machine learning packages use “automatic differentiation” (or “autograd”) to handle this, and with that name it sounds like everything just happens; as if you write the operations and the computer just figures out all the derivatives. But it can’t be this simple. In an operation such as convolution, a multiplicative sum is computed using a weight filter of some chosen size positioned at various locations over an input image, and the result is a new image in which each value corresponds to one such sum. How does “autograd” handle this?

Image for post

A 2x2 convolution: How do you take the derivative of that? (Figure by author)

Here we will:

  1. Show how backpropagation emerges from the “chain rule” of differentiation.
  2. Work through the computation of derivatives in some specific examples.__ We’ll use TensorFlow and aim to understand exactly what is computed when the `gradient_` function is called on a tensor._

tensorflow machine-learning artificial-intelligence

Bootstrap 5 Complete Course with Examples

Bootstrap 5 Tutorial - Bootstrap 5 Crash Course for Beginners

Nest.JS Tutorial for Beginners

Hello Vue 3: A First Look at Vue 3 and the Composition API

Building a simple Applications with Vue 3

Deno Crash Course: Explore Deno and Create a full REST API with Deno

How to Build a Real-time Chat App with Deno and WebSockets

Convert HTML to Markdown Online

HTML entity encoder decoder Online

AI(Artificial Intelligence): The Business Benefits of Machine Learning

Enroll now at CETPA, the best Institute in India for Artificial Intelligence Online Training Course and Certification for students & working professionals & avail 50% instant discount.

Learning in Artificial Intelligence - Great Learning

What is Artificial Intelligence (AI)? AI is the ability of a machine to think like human, learn and perform tasks like a human. Know the future of AI, Examples of AI and who provides the course of Artificial Intelligence?

Artificial Intelligence, Machine Learning, Deep Learning 

Artificial Intelligence (AI) will and is currently taking over an important role in our lives — not necessarily through intelligent robots.

How To Get Started With Machine Learning With The Right Mindset

You got intrigued by the machine learning world and wanted to get started as soon as possible, read all the articles, watched all the videos, but still isn’t sure about where to start, welcome to the club.

Hire Machine Learning Developers in India

We supply you with world class machine learning experts / ML Developers with years of domain experience who can add more value to your business.