What Autocomplete can do on a small scale, GPT-3 does on a much larger one. This is what makes it the Great Gatsby of chatbots. You simply give it a few words or sentences, and the language AI spits out a few pages of text based on just a few default settings.
The software is made up of a neural network comprising 175 billion parameters. The same principle applies to AI development: the more, the better. More data, including from a greater variety of sources, allow for more suitable responses to prompts.
After all, the machines never know anything about what they are saying. As far as they can tell, real news is just as true as fictional stories, and the actor Johnny Depp is just as real to it as Captain Jack Sparrow, the role he portrays in Pirates of the Caribbean.
However, these machines have been fed the entirety of Wikipedia, social media feeds, and reddit threads. Their answers sound amazingly human.
Yet computer scientist Yonatan Bisk is not impressed: "And nobody is surprised that if you memorize more, you can do more."
What the transformer can do in a nutshell:
- Generate a huge amount of text in just a few seconds
- Code programs on command, even though this falls outside of its original programming
This is where pre-training pays off. GPT-3 understands language commands that are entered as text and is proficient in programming languages.
This makes the language model a generalist, a true jack-of-all-trades, which does not have to be taught everything individually.
It is more adept at:
- Completing sentences
- Laying out images and designs
- Solving simple logic tasks, such as drawing analogies.
GPT-3 could even use its own code and create language model software “in its own image” (to highlight its omnipotence over all AIs).
Popular uses of the technology have already emerged for which GPT-3 has not been specifically programmed or trained. Interested parties with developer access to the prototype version can train it for specific applications, including as a chatbot, web design machine, program assistant, or intelligent search engine.
All the user needs to do is provide a prompt.
Start typing a prompt of the type "Chocolate cakes are...", and the transformer looks for the most common terms in its neural network that are used to describe chocolate cake and then spits out whole sentences and paragraphs. Due to the immense amount of data in its memory, the text remains largely coherent.
At least until the most frequent terms it finds deviate from the topic of the composition, resulting in the content of the text becoming more and more arbitrary and thus, at a certain point, plain factually wrong.
Or just try to give it an assignment like: "Write an essay about the development of the American colonies that became the US, and compile statistics about the population growth for all the colonies up to the election of the first US president."
Relying on all of the Wiki facts that it ingested, GPT-3 is not only able to write an essay, but it also generates a formatted table with the colonies and their population figures by year.
Philosophical wisdom can also be wrested from the language model by using a prompt like: "Dear human philosophers, I read your comments on my abilities and limitations with great interest." The full answer revolving around lying robots is surprisingly honest, making for an entertaining and worthwhile read.
GPT-3 still doesn't know what it's talking about, but it managed to generate the essay in just half a minute. Of course, there is also utter nonsense in it, because it uncontrollably combines very common terms into sentences, which the machine openly admits to doing in several paragraphs.
Often it works well, but it is just as often completely wrong. GPT-3 has not yet graduated from high school. Instead, the developers have placed their super-transformers under house arrest, and visits are only permitted under their strict supervision. Furthermore, GPT-3 is only available to entrepreneurs who apply to OpenAI with ethically sound ideas. The fear that the technology will be manipulated and abused is simply too great.