A Guide to Word Embeddings

Reading, comprehending, communicating and ultimately producing new content is something we all do regardless of who we are in our professional lives.

When it comes to extracting useful features from a given body of text, the processes involved are fundamentally different when compared to, say a vector of continuous integers. This is because the information in a sentence or a piece of text is encoded in structured sequences, with the semantic placement of words conveying the meaning of the text.

So this dual requirement of appropriate representation of the data along with preserving the contextual meaning of the text has led me to learn about and implement 2 different NLP models to achieve the task of text classification.

Word Embeddings are dense representations of the individual words in a text, taking into account the context and other surrounding words that that individual word occurs with.

The dimensions of this real-valued vector can be chosen and the semantic relationships between words are captured more effectively than a simple Bag-of-Words Model.

#keras #nlp #data-science #machine-learning #questions

What is GEEK

Buddha Community

Word Embeddings Versus Bag-of-Words: The Curious Case of Recommender Systems

Are word embeddings always the best choice?

If you can challenge a well-accepted view in data science with data, that’s pretty cool, right? After all, “in data we trust”, or so we profess! Word embeddings have caused a revolution in the world of natural language processing, as a result of which we are much closer to understanding the meaning and context of text and transcribed speech today. It is a world apart from the good old bag-of-words (BoW) models, which rely on frequencies of words under the unrealistic assumption that each word occurs independently of all others. The results have been nothing short of spectacular with word embeddings, which create a vector for every word. One of the oft used success stories of word embeddings involves subtracting the man vector from the king vector and adding the woman vector, which returns the queen vector:

Very smart indeed! However, I raise the question whether word embeddings should always be preferred to bag-of-words. In building a review-based recommender system, it dawned on me that while word embeddings are incredible, they may not be the most suitable technique for my purpose. As crazy as it may sound, I got better results with the BoW approach. In this article, I show that the uber-smart feature of word embeddings in being able to understand related words actually turns out to be a shortcoming in making better product recommendations.

Word embeddings in a jiffy

Simply stated, word embeddings consider each word in its context; for example, in the word2vec approach, a popular technique developed by Tomas Mikolov and colleagues at Google, for each word, we generate a vector of words with a large number of dimensions. Using neural networks, the vectors are created by predicting for each word what its neighboring words may be. Multiple Python libraries like spaCy and gensim have built-in word vectors; so, while word embeddings have been criticized in the past on grounds of complexity, we don’t have to write the code from scratch. Unless you want to dig into the math of one-hot-encoding, neural nets and complex stuff, using word vectors today is as simple as using BoW. After all, you don’t need to know the theory of internal combustion engines to drive a car!

#cosine-similarity #bag-of-words #python #word-embeddings #recommendation-system

Introduction to Word Embeddings

Word embedding is a method to capture the “meaning” of the word via low dimension vector and it can be used in a variety of tasks in Natural Language Processing (NLP).

Before beginning word embedding tutorial we should have an understanding of vector space and similarity matrix.

Vector Space

A sequence of numbers that is used to identify a point in space is called vector and if we have a whole bunch of vectors that all belong to the same dataset it will be called a** vector space**.

Words in the text can also be represented in the higher dimension in vector space where words having the same meaning will have similar representations. For example,

photo by Allision Parrish from Github

The above image shows a vector representation of words on the scale of cuteness and size of animals. we can see that there is a semantic relationship between words on bases of similar properties. It is difficult to represent the higher dimensional relationship between words but the maths behind is the same so it works similarly in a higher dimension also.

Similarity matrix

It is used to calculate the distance between vectors in the vector space. it measures similarity or distance between two data points in vector space. This allows us to capture words that are used in similar ways to result in having similar representation naturally capturing their meaning. there is a lot of similarity matrix available but we will discuss Euclidean distance and Cosine similarity.

Euclidean distance

One way to calulate how far two data points are in vector space is to calculate Euclidean distance.

import math
def distance2d(x1, y1, x2, y2):
    return math.sqrt((x1 - x2)**2 + (y1 - y2)**2)

So, the distance between “capybara” (70, 30) and “panda” (74, 40) from the above image example:

… is less than the distance between “tarantula” and “elephant” from the above image example:

This shows that “pandas” and “capybara” are more similar as compared to “tarantula” and “elephant”.

Cosine similarity

It is a measure of similarity between two non-zero vectors of an inner product space that measures the cosine of the angle between them.

from numpy import dot
from numpy.linalg import norm

cos_sim = dot(a, b)/(norm(a)*norm(b))

#data-science #machine-learning #deep-learning #word-embeddings #nlp #deep learning

Word Embedding Fairness Evaluation

Word embeddings are dense vector representations of words trained from document corpora. They have become a core component of natural language processing (NLP) downstream systems because of their ability to efficiently capture semantic and syntactic relationships between words. A widely reported shortcoming of word embeddings is that they are prone to inherit stereotypical social biases exhibited in the corpora on which they are trained.

The problem of how to quantify the mentioned biases is currently an active area of research, and several different fairness metrics have been proposed in the literature in the past few years.

Although all metrics have a similar objective, the relationship between them is by no means clear. Two issues that prevent a clean comparison is that they operate with different inputs (pairs of words, sets of words, multiple sets of words, and so on) and that their outputs are incompatible with each other (reals, positive numbers,  range, etc.). This leads to a lack of consistency between them, which causes several problems when trying to compare and validate their results.

We propose the Word Embedding Fairness Evaluation (WEFE) as a framework for measuring fairness in word embeddings, and we released its implementation as an open-source library.

Framework

We propose an abstract view of a fairness metric as a function that receives queries as input, with each query formed by a target and attribute words. The target words describe the social groups in which fairness is intended to be measured (e.g., women, white people, Muslims), and the attribute words describe traits or attitudes by which a bias towards one of the social groups may be exhibited (e.g., pleasant vs. unpleasant terms). For more details on the framework, you can read our recently accepted paper IJCAI paper [1].

WEFE implements the following metrics:

• Word Embedding Association Test (WEAT)
• Relative Norm Distance (RND)
• Relative Negative Sentiment Bias (RNSB)
• Mean Average Cosine (MAC)

#bias #ethics #machine learning #word embeddings

Help Your Data Science Career By Publishing Your Work!

This guide aims to cover everything that a data science learner may need to write and publish articles on the internet. It covers why you should write, writing advice for new writers, and a list of places that invite contributions from new writers.

Let’s get to it!

Why you should write:

Writing isn’t just for “writers”. The art of writing well is for everyone to learn - programmers, marketers, managers and leaders, alike. And yes, data scientists and analysts too!

You should write articles because when you do:

You learn:

Writing teaches you the art of writing. It’s kind of circular but it’s true.

Make no mistake, the art of writing isn’t about grammar (although, that’s important) and flowery language (definitely not important). It’s about conveying your thoughts with clarity in simple language.

And learning this art is important even if you absolutely know that you don’t want to write blogs/articles for a living. It’s important because all the jobs have some form of writing involved - messages, emails, memos and the whole spectrum. So basically, writing is a medium for almost any job you can have.

Apart from that, when you write you learn the things that you thought you knew but didn’t really know. So, writing is an opportunity to learn better.

#data science career tips #guide #guides #publishing work #writing guide

Introduction to Word Embeddings (NLP)

In other words, we want to find an embedding for each word in some vector space and we wanted to exhibit some desired properties.

Representation of different words in vector space (Image by author)

For example, if two words are similar in meaning, they should be closer to each other compared to words that are not_. And, if two pair of words have a similar difference in their meanings, _they should be approximately equally separated in the embedded space.

We could use such a representation for a variety of purposes like finding synonyms and analogies, identifying concepts around which words are clustered, classifying words as positive, negative, neutral, etc. By combining word vectors, we can come up with another way of representing documents as well.

---

Word2Vec — The General Idea

Word2Vec is perhaps one of the most popular examples of word embeddings used in practice. As the name Word2Vec indicates, it transforms words to vectors. But what the name doesn’t give away is how that transformation is performed.

#machine-learning #data-science #nlp #word-embeddings #artificial-intelligence