As a professor of innovation, there’s a slide in my deck I use every time I start a session on artificial intelligence: a fragment of the first page from the “Attention Is All You Need” paper, from 2017. I use just a fragment because I don’t want the students to read it… I usually refrain from using a lot of academic papers when I teach MBAs and executives: they tend to be boring, and this one is no exception and is not particularly easy to read. However, it is historic. Written by eight Google researchers, it is the paper that defines the Transformer architecture, which among other things, became the architecture behind GPT and much of the subsequent generative-AI wave. The rest, as I said, is history.

The Transformer was not only a successful model architecture: it was also a huge architectural discontinuity. The 2017 breakthrough was able to completely reorganize computation around self-attention, to eliminate recurrence and to make parallel training much easier. And in doing so, it did not simply enlarge what was already there: it completely changed the architecture. 

In 2017, Google understood the technology, but underestimated the category it could create. It did not ignore the Transformer: it incorporated the architecture into products such as Translate and Search, but it failed to turn its own breakthrough into the product that defined the next era. OpenAI , founded two years before the paper appeared, recognized its broader potential and built GPT, and ultimately ChatGPT , on top of it.

Source link