How to Apply Transformers to Any Length of Text

Restore the power of NLP for long sequences

https://towardsdatascience.com/how-to-apply-transformers-to-any-length-of-text-a5601410af7f

Friend link (free access)

The de-facto standard in many natural language processing (NLP) tasks nowadays is to use a transformer. Text generation? Transformer. Question-and-answering? Transformer. Language classification? Transformer!

However, one of the problems with many of these models (a problem that is not just restricted to transformer models) is that we cannot process long pieces of text.

Almost every article I write on Medium contains 1000+ words, which, when tokenized for a transformer model like BERT, will produce 1000+ tokens. BERT (and many other transformer models) will consume 512 tokens max — truncating anything beyond this length.

Although I think you may struggle to find value in processing my Medium articles, the same applies to many useful data sources — like news articles or Reddit posts.

We will take a look at how we can work around this limitation. In this article, we will find the sentiment for long posts from the /r/investing subreddit. This article will cover:

• High-Level Approach
• Getting Started
  • Data
  • Initialization
• Tokenization
• Preparing The Chunks
  • Split
  • CLS and SEP
  • Padding
  • Reshaping For BERT
• Making Predictions

python nlp deep-learning machine-learning tensorflow pytorch

PyTorch for Deep Learning | Data Science | Machine Learning | Python. PyTorch is a library in Python which provides tools to build deep learning models. What python does for programming PyTorch does for deep learning. Python is a very flexible language for programming and just like python, the PyTorch library provides flexible tools for deep learning.

We will go over what is the difference between pytorch, tensorflow and keras in this video. Pytorch and Tensorflow are two most popular deep learning frameworks. Pytorch is by facebook and Tensorflow is by Google. Keras is not a full fledge deep learning framework, it is just a wrapper around Tensorflow that provides some convenient APIs.

In this video I am discussing various techniques to handle imbalanced dataset in machine learning. I also have a python code that demonstrates these different techniques. In the end there is an exercise for you to solve along with a solution link. Credit card fraud detection, cancer prediction, customer churn prediction are some of the examples where you might get an imbalanced dataset. Training a model on imbalanced dataset requires making certain adjustments otherwise the model will not perform as per your expectations.

PyTorch is a library in Python which provides tools to build deep learning models. What python does for programming PyTorch does for deep learning.

PyTorch Tutorial - Deep Learning Using PyTorch - Learn PyTorch from Basics to Advanced. Learn PyTorch from the very basics to advanced models like Generative Adverserial Networks and Image Captioning. "PyTorch: Zero to GANs" is an online course and series of tutorials on building deep learning models with PyTorch, an open source neural networks library.