# Reflections on the Influence of Transformers and ChatGPT: Perspectives from Jakob Uszkoreit
In 2017, a group of eight machine-learning experts at Google released a pioneering paper named *Attention Is All You Need*. This publication presented the *Transformer* architecture, a neural network framework that has since laid the groundwork for many of today’s leading AI models, such as OpenAI’s ChatGPT and various AI-focused tools from Google. One of the co-authors of this influential work, Jakob Uszkoreit, recently shared his thoughts on the evolution of the Transformer, the emergence of ChatGPT, and Google’s measured strategy towards AI advancement during an interview with Ars Technica at TED AI 2024.
## The Origin of the Transformer
The Transformer architecture transformed the domain of natural language processing (NLP) by superseding the previously prevalent Recurrent Neural Networks (RNNs) with a method known as *self-attention*. This breakthrough enabled AI models to handle data more efficiently and competently, resulting in remarkable progress in understanding and generating language.
Uszkoreit, who was instrumental in advocating for the application of self-attention, mentioned that the team’s work was founded on the contributions of numerous prior research initiatives. “It wasn’t just that one paper,” he remarked, highlighting the collaborative and progressive character of scientific endeavors. Although the team had lofty expectations for the technology, they did not fully foresee the degree to which it would facilitate products like ChatGPT.
## ChatGPT: An Unexpected Public Triumph
When OpenAI unveiled ChatGPT in late 2022, it rapidly enchanted the public, morphing into a cultural sensation. Uszkoreit conceded that while his colleagues at Google had spotted early indicators of the promise of large language models (LLMs), they did not anticipate the remarkable success of ChatGPT. “We observed phenomena that were quite astonishing,” he stated, yet Google’s cautious product development philosophy at the time meant these potentials were not entirely realized in consumer-oriented products.
Uszkoreit reminisced about the time when ChatGPT became a public hit, recognizing a feeling of “Whoa, that could have happened sooner.” However, he was even more struck by how swiftly people adapted to the technology and discovered inventive ways to utilize it. “That was just breathtaking,” he commented.
## Google’s Careful Strategy
During the interview, Uszkoreit discussed Google’s prudent approach to launching AI products like ChatGPT. He clarified that although Google possessed the capabilities to develop similar models, the organization was reluctant to embrace risks. This caution, he asserted, might have postponed the introduction of groundbreaking AI products.
Uszkoreit drew a comparison between ChatGPT and Google Translate, a project he had contributed to for many years. At its inception, Google Translate was far from flawless, frequently generating amusingly inaccurate translations. Nevertheless, over the years, it matured into a highly effective tool. “Google did it anyway because it was the right thing to try,” he expressed, noting that this eagerness to explore was more common in the early days of Google.
## The Significance of Experimentation
One of the pivotal insights from Uszkoreit’s reflections is the critical role of experimentation and willingness to take risks in AI development. He underscored that breakthrough innovations often arise from deploying technology in practical settings and learning from user interactions. “You always have to take into account how your users actually use the tool that you create,” he stated, adding that users’ ingenuity in discovering new applications can sometimes only emerge through real-world experimentation.
Uszkoreit also emphasized the necessity for companies to stay “experiment-happy” and “failure-happy,” as most trials may not yield successful outcomes. Nonetheless, the rare triumphs, like ChatGPT, can significantly impact the landscape.
## A New Era: Biological Computing
After departing Google, Uszkoreit co-founded *Inceptive*, a firm dedicated to leveraging deep learning in biochemistry. Inceptive is working on what Uszkoreit refers to as “biological software,” wherein AI compilers convert specified behaviors into RNA sequences that can execute particular functions within biological systems.
Uszkoreit elucidated that this method is akin to how conventional software is compiled for computers. However, instead of employing an engineered compiler, Inceptive utilizes a learned AI compiler to create molecules that can demonstrate complex behaviors when introduced into living organisms. He cited mRNA COVID-19 vaccines as a straightforward illustration of biological software, where the RNA program instructs cells to generate a viral antigen.
Looking ahead, Uszkoreit foresees a future in which more intricate biological programs can be devised to address illnesses and enhance human health. “If we managed to even just design molecules with a teeny tiny fraction of such functionality, it would truly transform medicine,” he remarked.
## The Future of AI and Medicine
As AI continues to advance, its implications in sectors like medicine promise tremendous potential. Uszkoreit believes that AI-driven biological software could redefine the way we engage with healthcare, opening new frontiers in treatment and wellness.
