A lot of ink has been spilled (or pixels illuminated) about the wonders of GPT-3, OpenAI’s latest and greatest language model. In the words of Venturebeat:

A team of more than 30 OpenAI researchers have released a paper about GPT-3, a language model capable of achieving state-of-the-art results on a set of benchmark and unique natural language processing tasks that range from language translation to generating news articles to answering SAT questions.

And as tweeted by MIT:

To be fair, some of the examples are amazing:

But like most examples spat out by language models, almost all of these were hand-selected by humans after many runs. Because not-so-good results just wouldn’t make the news.

Even bearing that in mind, I’m still blown away by what I’ve seen of GPT-3. But right now there’s no way I can get my hands on it for any practical purposes — it’s only available to a few researchers at the moment. What about us mere mortals? Other than playing AI Dungeon of course…

Play with GPT-2!

For now, we can play with GPT-3’s older brother, GPT-2. A few weeks ago I wrote easy_text_generator, a tool to generate text of any length from language models in your browser. You can use GPT-2 or a few pre-trained or distilled versions of it (downloaded from huggingface) to generate different kinds of text, like movie scripts, Star Trek scripts, IMDB movie reviews, or just text in general.

Or if you want something with zero install, you can generate text directly from huggingface’s site, just with limited length. For example, try out GPT-2 here.

Let’s Generate Some Text!

As I said, most examples of GPT-3 you may have seen have been cherry-picked. To make a reasonably fair comparison, I’ll run each language model 5 times and cherry-pick one output for each model. The other ones I’ll put in another post, otherwise, this blog post will get too long.

Let’s look at general purpose text at first. Since I just started working at a company doing a neural search, we’ll use some appropriate seed text:

_In short, neural search is a new approach to retrieving information._

GPT-2

GPT-2 can create amazing results (check out the English speaking unicorns example) but it takes its sweet time and gobbles up CPU and memory. Here’s what it spat out after about 10 minutes:

In short, neural search is a new approach to retrieving information. In an ideal world, there would be lots of new data to search, but most of it is already stored in a relational database. That’s why most of the time it’s best not to store any information in the database at all. And in the future, what we want is that the information is simply searchable. But even if the information is readily searchable, there is still the problem of how to combine multiple search terms in the database. One way to combine search terms is to have search engines, but when we search for something on Google or any other search engine, they can often be quite good, but sometimes they’ll make mistakes, and they’ll only show some of the results, and sometimes they won’t show all of the results or they’ll say something very specific but you can’t look it up in that particular search. One of the most important goals of neural search is to let them search even more easily so they can make better decisions. But there’s still one issue: some types of problems don’t lend themselves to it and this can lead to a lot of lost information if the search engine doesn’t have the right tool in the right place. Nurture Now, I think we can take some lessons from neural search to solve this problem…

Looks somewhat coherent to me, though it doesn’t keep a clear narrative going, and Dear Lord, some of those run-on sentences!

#deep-learning #artificial-intelligence #gpt-3 #machine-learning #nlp #deep learning