In the last post I really discussed GANs (the structure and the steps of the training and the loss function). In this post, I will discuss the limitation and the problem of training a GAN.

In **the last post** I really discussed GANs (the structure and the steps of the training and the loss function). In this post, I will discuss **the limitation and the problem of training a GAN**.

There are many various problems that prevent successful GAN training due to the GANsâ€™ extremely diverse applications. Accordingly, improving the training of the GANs is an open research field for researchers.

Before discussing the problems, let have a quick look at some of the GAN equations.

The architecture of the GAN with derivation terms of the loss function

The figure above indicates, the derivation terms of the final loss function for training the discriminator and generator with the corresponding gradient. In the following are some of the basic problems of GAN that lead to hard training.

The first problem in the GAN training that should be considered serious is **vanishing gradients**. Before diving into the Vanishing gradients problem, the **KL-divergence** and **JS-divergence** should be explained.

The concept of divergence between two probability distributions can be defined as **a measure of the distance between two distributions**. Imagine if we minimize the divergence, we also hope that the two distributions are equal.

In many generative models, the goal is to create a model which maximizes the **Maximum Likelihood Estimation**(shortly MLE) which can be defined as the best model parameters to fit the training data. Maximum likelihood equation is shown in follow.

kl-and-js-divergence mode-collapse nash-equilibrium problem-of-gan vanishing-gradient neural networks

Understanding the obstacles that faces us when building deep neural networks. A difficulty that we are faced with when training deep Neural Networks is that of vanishing or exploding gradients.

In this post, we will learn how to create a self-normalizing deep feed-forward neural network using Keras. This will solve the gradient instability issue, speeding up training convergence, and improving model performance.

There has been hype about artificial intelligence, machine learning, and neural networks for quite a while now. This will not be a math-heavy introduction because I just want to build the idea here.

Neural networks, as their name implies, are computer algorithms modeled after networks of neurons in the human brain. Learn more about neural networks from Algorithmia.

Recurrent neural networks, also known as RNNs, are a class of neural networks that allow previous outputs to be used as inputs while having hidden states.