One of the very popular ways to structure a neural network is called a Convolutional Neural Network. It was invented by Yann LeCun about 30 years ago, but it’s become incredibly popular for things like image processing, and processing large datasets. So, let’s talk about Convolutional Neural Networks.

Photo by Victor Grabarczyk on Unsplash

Statistical Invariance

Here’s an example. We have an image, and we want our network to say it’s an image with a cat in it. It doesn’t really matter where the cat is, it’s still an image with a cat. If our network has to learn about kittens in the left corner, and about kittens in the right corner independently, that’s a lot of work that it has to do. How about we telling it, instead explicitly, that objects and images are largely the same whether they’re on the left or on the right of the picture.

That’s what’s called translation invariance. Different positions, same kitten. Yet another example. Imagine we have a long text that talks about kittens. Does the meaning of kitten change depending on whether it’s in the first sentence or in the second one? Mostly not, so if we’re trying to network on text, maybe we want the part of the network that learns what a kitten is to be reused every time we see the word kitten, and not have to re-learn it every time. The way we achieve this in our own networks, is using what’s called weight sharing. When we know that two inputs can contain the same kind of information, then we share their weights. And train the weights jointly for

those inputs. It’s a very important idea. Statistical invariants, things that don’t change on average across time or space, are everywhere. For images, the idea of weight sharing will get us to study cnn’s. For text and sequences in general, it will lead us to embeddings and recurrent neural networks.

#self-driving-cars #deep-learning #artificial-intelligence #data-science #machine-learning