Refresher on one-hot vectors

Remember one-hot vectors? No? Well do you remember unit vectors from math class? Still no? Okay — assume that we have three labels, [🍎, 🍊, 🍌]. We want to represent these values in a way that machines can understand. Initially, we might be tempted to assign the values [1, 2, 3], but the issue here is we don’t necessarily want a 🍌 to equal three 🍎.

We could instead assign vectors to each label, where the dimension of each vector is equal to the number of labels. In this case we have three labels, so three dimensions.

Embeddings

Think about a company that has a million books in that company’s catalogue, which they would like to use as input data. It is not practical to create a system that needs a million-dimension vector to represent each book. Unless they are specifically looking for something unique to each book, there must be some aspect that some books will share. How can we manage that?

With one-hot vectors, each vector is unique, and the dot product of any two vectors is always 0. However, what if this is not desired and we want there to be some similarity between vectors (a non-zero dot product)? Going back to our example above, what if in our application we are looking at the shapes of fruit? It is logical that there would be some overlap of values.

#artificial-intelligence #introduction #categorical-data #embedding-layer #machine-learning #deep learning