Can Unconditional Language Models Recover Arbitrary Sentences? — A paper summary

Can Unconditional Language Models Recover Arbitrary Sentences? — A paper summary

This article is a summary of this paper by researchers at New York University which tries to ascertain exactly this, whether these models can recover an arbitrary sentence from its encoded representation.

These days BERTELMo and Ernie reminds one of pre-trained generative models rather than Sesame Street characters, such has been their hegemony over the Natural Language Processing landscape. These models can serve as general purpose encoders, and can even perform some tasks like text classification without requiring further modification. However, limited research has been conducted on the reverse-case, exploiting these models for use as general purpose decoders. This article is a summary of this paper by researchers at New York University which tries to ascertain exactly this, whether these models can recover an arbitrary sentence from its encoded representation.

Abstract

In order to prove the existence of encoded representations that can be used for recovering a sentence, the paper introduces methods to feed these representations into a recurrent language model trained autoregressivelyas well as map sentences into and out of this “reparametrized” space, while keeping the main language model parameters frozen.

Method

Recap on Recurrent Language Models

Before we begin, let us quickly look at recurrent language models and how they can be trained autoregressively.

Image for post

The mathematical representation of an autoregressive language model. (Source: paper)

Recall that in an autoregressive model, we take as input all the previous tokens, combine it with the previous hidden state and compute the next token. This hidden state is often implemented as a LSTMrecurrent network, and the final output is nothing but a softmaxfunction, which indicates the dedicated probability of a particular word being the next token.

data-science machine-learning nlp recurrent-neural-network

Bootstrap 5 Complete Course with Examples

Bootstrap 5 Tutorial - Bootstrap 5 Crash Course for Beginners

Nest.JS Tutorial for Beginners

Hello Vue 3: A First Look at Vue 3 and the Composition API

Building a simple Applications with Vue 3

Deno Crash Course: Explore Deno and Create a full REST API with Deno

How to Build a Real-time Chat App with Deno and WebSockets

Convert HTML to Markdown Online

HTML entity encoder decoder Online

Artificial Neural Networks — Recurrent Neural Networks

Artificial Neural Networks — Recurrent Neural Networks. Remembering the history and predicting the future with neural networks. A intuition behind Recurrent neural networks.

Most popular Data Science and Machine Learning courses — July 2020

Most popular Data Science and Machine Learning courses — August 2020. This list was last updated in August 2020 — and will be updated regularly so as to keep it relevant

Cheat Sheets for AI, Neural Networks, Machine Learning, Deep Learning & Big Data

Cheat Sheets for AI, Neural Networks, Machine Learning, Deep Learning & Big Data

15 Machine Learning and Data Science Project Ideas with Datasets

Learning is a new fun in the field of Machine Learning and Data Science. In this article, we’ll be discussing 15 machine learning and data science projects.

Fundamentals of Neural Network in Machine Learning

Fundamentals of Neural Network in Machine Learning. What is a Neuron? What is the Activation Function? How do Neural Network Works? How do Neural Networks Learn?