GPT-3 is likely the most computationally-expensive machine learning model. The neural network’s 175 billion parameters make it about ten times larger than the previous largest language model (Turing NLG, 17 billion parameters, released by Microsoft in February 2020). The 430GB of text GPT-3 was trained on was drawn widely from the internet and supplemented with text from books. The model works by seeing some amount of text that has come previously (up to a maximum of about 2,000 words) and predicting the next word to generate novel text.

#nlp #ai #openai #machine-learning #gpt-3