This article is a summary of this paper by researchers at New York University which tries to ascertain exactly this, whether these models can recover an arbitrary sentence from its encoded representation.
These days BERT, ELMo and Ernie reminds one of pre-trained generative models rather than Sesame Street characters, such has been their hegemony over the Natural Language Processing landscape. These models can serve as general purpose encoders, and can even perform some tasks like text classification without requiring further modification. However, limited research has been conducted on the reverse-case, exploiting these models for use as general purpose decoders. This article is a summary of this paper by researchers at New York University which tries to ascertain exactly this, whether these models can recover an arbitrary sentence from its encoded representation.
In order to prove the existence of encoded representations that can be used for recovering a sentence, the paper introduces methods to feed these representations into a recurrent language model trained autoregressivelyas well as map sentences into and out of this “reparametrized” space, while keeping the main language model parameters frozen.
Before we begin, let us quickly look at recurrent language models and how they can be trained autoregressively.
The mathematical representation of an autoregressive language model. (Source: paper)
Recall that in an autoregressive model, we take as input all the previous tokens, combine it with the previous hidden state and compute the next token. This hidden state is often implemented as a LSTMrecurrent network, and the final output is nothing but a softmaxfunction, which indicates the dedicated probability of a particular word being the next token.
Artificial Neural Networks — Recurrent Neural Networks. Remembering the history and predicting the future with neural networks. A intuition behind Recurrent neural networks.
Most popular Data Science and Machine Learning courses — August 2020. This list was last updated in August 2020 — and will be updated regularly so as to keep it relevant
Cheat Sheets for AI, Neural Networks, Machine Learning, Deep Learning & Big Data
Learning is a new fun in the field of Machine Learning and Data Science. In this article, we’ll be discussing 15 machine learning and data science projects.
Fundamentals of Neural Network in Machine Learning. What is a Neuron? What is the Activation Function? How do Neural Network Works? How do Neural Networks Learn?