The Serendipitous Success of ChatGPT
Ever since the launch of ChatGPT in Nov 2022, many people have asked what I think about this innovation. My response has always been consistent in that although it’s a very cool technology, there is really nothing innovative about it. And after 5 months, I still hold true to my stance. If there is anything new and different about the launch of ChatGPT, it’s the public that has come to the realization of the power of AI. But in terms of the science and algorithm, there really isn’t that much novelty.
Is ChatGPT Really an Innovation?
The GPT model underlying ChatGPT is a kind of deep neural network called transformer, hence the name Generative Pre-trained Transformer (GPT). However, OpenAI didn’t invent the transformer architecture. In fact, it was invented by Google through the seminal paper “Attention is All You Need.” At the time, it was a pretty novel approach to language modeling, because it uses a feed-forward neural network as opposed to the more conventional recurrent neural network architecture. But this was way back in 2017, and Google has already used transformers in many of its products (e.g. BERT, Google Translate, Google Assistant, and even it’s flagship Google Search).
OpenAI simply scaled up the transformer (i.e. 175 billion parameters in GPT-3.5 and over a trillion parameters in GPT-4) and trained it with a huge number of documents. This is undeniably a colossal engineering feat!
However, the GPT model is initially pre-trained via supervised learning to predict the next word based on the preceding words as in any language model. This pre-trained large language model (LLM) is then fine-tuned using transfer learning to produce responses that mimic human dialogues (i.e. chatting). Finally, ChatGPT is further fine-tuned via human-in-the-loop reinforcement learning to output the most desirable response according to the reward schedule specified by human rankings. Yet, none of these techniques are considered technological breakthroughs in anyways.
Getting the Human Factor Right
So how was ChatGPT able to create such a tidal wave of awareness?
If there is anything that OpenAI got right, it’s the verbosity level for the textual chat interaction. Since ChatGPT doesn’t have a voice, it can be much more verbose. This is crucial to its expressiveness, which connects with users on a human level. Unlike other language models that can feel cold and impersonal, ChatGPT has a natural and engaging conversational style that makes it feel like a real person.
The lack of a voice is actually advantageous here, because users can fill in the proper voice in their mind based on the context of the dialogue as if they are reading a novel. This level of verbosity would not work for digital assistants that already have a rather rigid robotic voice. Imagine how annoying it would be if Siri or Alexa is only half as verbose as ChatGPT. It would drive me nuts!
Conclusion
IMHO, besides the engineering achievement that scaled up the GPT model to trillions of parameters, all the technologies behind ChatGPT already exist. There is nothing preventing Google, Meta, or any other tech giants from building a verbose chatbot from their own variant of LLM. But ChatGPT’s verbosity is trained and perfected for the chat medium, which creates engaging human-like conversations. This has made it popular not just among tech enthusiasts, but also among casual users who appreciate its friendly and approachable demeanor.
Was this planned? Maybe…, but it may also be completely accidental.
Now, I’m not saying that ChatGPT is not an amazing tool. It certainly is! But you don’t need to invent something completely novel to be amazing. In fact, the most successful application of any technology rarely comes from its inventors. Before Google, there are actually dozens of web search engines. Likewise, before ChatGPT, there are already many existing LLM-based chatbots. For OpenAI, getting the human factor right is the key to its meteoric rise.
Want to go beyond ChatGPT and get more insights about Generative AI? You should attend my keynote at Outperform with PROS – Beyond ChatGPT: Inside the Minds of Generative AI.