Accelerate analysis by bringing structure to unstructured data

Working with natural language data can often be challenging due to its lack of structure. Most data scientists, analysts and product managers are familiar with structured tables, consisting of rows and columns, but less familiar with unstructured documents, consisting of sentences and words. For this reason, knowing how to approach a natural language dataset can be quite challenging. In this post I want to demonstrate how you can use the awesome Python packages, spaCy and Pandas, to structure natural language and extract interesting insights quickly.

Introduction to Spacy

spaCy is a very popular Python package for advanced NLP — I have a beginner friendly introduction to NLP with SpaCy here. spaCy is the perfect toolkit for applied data scientists when working on NLP projects. The api is very intuitive, the package is blazing fast and it is very well documented. It’s probably fair to say that it is the best general purpose package for NLP available. Before diving into structuring NLP data, it is useful to get familiar with the basics of the spaCy library and api.

After installing the package, you can load a model (in this case I am loading the simple Engilsh model, which is optimized for efficiency rather than accuracy) — i.e. the underlying neural network has fewer parameters.

import spacy
nlp = spacy.load("en_core_web_sm")

We instantiate this model as nlp by convention. Throughout this post I’ll work with this dataset of famous motivational quotes. Let’s apply the nlp model to a single quote from the data and store it in a variable.

#analytics #nlp #machine-learning #data-science #structured natural language processing with pandas and spacy #natural language processing