Deep learning basics — batch normalization

Deep learning basics — batch normalization

What is batch normalization? This article explains batch normalization in a simple way. I wrote this article after what I learned from Fast.ai and deeplearning.ai.

What is batch normalization?

Batch normalization normalizes the activations of the network between layers in batches so that the batches have a mean of 0 and a variance of 1. The batch normalization is normally written as follows:

Image for post

https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm2d.html

The mean and standard-deviation are calculated per-dimension over the mini-batches and γ and β are learnable parameter vectors of size C (where C is the input size). By default, the elements of γ are set to 1 and the elements of β are set to 0.(https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm2d.html)

The mean and standard deviation are calculated for each batch and for each dimension/channel. γ and β are learnable parameters which can be used to scale and shift the normalized value, so that we can control the shape of the data when going into the next layer (e.g., control the percentage of positive and negative values going into a ReLU).

Ideally we would do this activation normalization for the entire dataset, however, it is often not possible due to the large size of the data. Thus, we try do to the normalization for each batch. Note that we prefer to have large batch sizes. If the batch size is too small, the mean and standard deviation would be very sensitive to outliers. If our batch sizes are large enough, the mean and standard deviations would be more stable.

pytorch batch-normalization python data-science deep-learning

Bootstrap 5 Complete Course with Examples

Bootstrap 5 Tutorial - Bootstrap 5 Crash Course for Beginners

Nest.JS Tutorial for Beginners

Hello Vue 3: A First Look at Vue 3 and the Composition API

Building a simple Applications with Vue 3

Deno Crash Course: Explore Deno and Create a full REST API with Deno

How to Build a Real-time Chat App with Deno and WebSockets

Convert HTML to Markdown Online

HTML entity encoder decoder Online

PyTorch for Deep Learning | Data Science | Machine Learning | Python

PyTorch for Deep Learning | Data Science | Machine Learning | Python. PyTorch is a library in Python which provides tools to build deep learning models. What python does for programming PyTorch does for deep learning. Python is a very flexible language for programming and just like python, the PyTorch library provides flexible tools for deep learning.

PyTorch for Deep Learning | Data Science | Machine Learning | Python

PyTorch is a library in Python which provides tools to build deep learning models. What python does for programming PyTorch does for deep learning.

Applied Data Science with Python Certification Training Course -IgmGuru

Master Applied Data Science with Python and get noticed by the top Hiring Companies with IgmGuru's Data Science with Python Certification Program. Enroll Now

Data Science Projects | Data Science | Machine Learning | Python

Practice your skills in Data Science with Python, by learning and then trying all these hands-on, interactive projects, that I have posted for you.

Data Science Projects | Data Science | Machine Learning | Python

Practice your skills in Data Science with Python, by learning and then trying all these hands-on, interactive projects, that I have posted for you.