Introduction — What is Zero-Shot Learning?

Zero-shot learning allows a model to recognize what it hasn’t seen before.

Imagine you’re tasked with designing the latest and greatest machine learning model that can classify all animals. Yes, all animals.

Using your machine learning knowledge, you immediately understand that we need a labeled dataset with at least one example for every single animal. There’s 1,899,587 described species in the world, so you’re gonna need a dataset with roughly 2 million different classes.

Yikes.

As you’ve probably noticed by now, getting large quantities of high quality labeled data is hard. Very hard.

It doesn’t help when there are a gazillion different classes (i.e. animal species) that your model has to learn.

So how do we solve this problem?

One way is to decrease our models’ reliance on labeled data. This is the motivation behind zero-shot learning,_ in which your model learns how to classify classes that it hasn’t seen before._

In the animal species classification example, your model may be able to predict that the image on the bottom right corner is a “Panda”, even though it didn’t explicitly see a labeled example of a “Panda” during training.

Crazy huh?!

In the next section, we’ll learn how this seemingly magical method works through some examples of models that employ a zero-shot setup.

How does Zero-Shot Learning Work?

Although there are multiple approaches to zero-shot learning in literature, this article focuses on a recent method called Contrastive Language-Image Pretraining (CLIP) proposed by OpenAI that has performed well in a zero-shot setting [2].

Just like traditional supervised models, CLIP has two stages: the training stage and the zero-shot inference stage.

In the training stage, CLIP learns about images by “reading” auxiliary text (i.e. sentences) corresponding to each image like in the example below.

#contrastive-learning #deep-learning #machine-learning #zero-shot-learning #artificial-intelligence