What is a transformer?

Type "I sat on the bank and watched the river" into a translation app, and it usually picks the word for a riverside, not the one for a place that keeps your money. Nobody told it which bank you meant. It worked that out from the other words in the sentence. The language models behind today's artificial intelligence (AI) chatbots do the same thing at enormous scale, and the design that made it practical is called a transformer.

Plenty of words mean nothing definite on their own. "Bank" could be money or mud. "Light" could be a lamp or a weight. Even "it" means nothing until you know what came before. People sort this out without noticing. For computers it was a hard problem for a long time, because older designs read one word after another and could lose track of what came much earlier in a long passage.²

The transformer's answer is a step called attention. For each word, the model looks across the other words in the passage and works out which ones matter most for that word's meaning. In our sentence, "bank" would weigh "river" and "watched" heavily, and "I" hardly at all. Then the model updates its sense of "bank" with what it found. It does this for every word, in many layers stacked one on top of another.²³

Attention brought a second advantage. Older designs had to work through a sentence in order. A transformer can look at every word in a passage at the same time, so the work can be split across many chips running side by side. Google's researchers reported in 2017 that it trained up to ten times faster than earlier designs on their translation tests. Speed like that is what made training on enormous amounts of text practical.¹³

The transformer was introduced in June 2017 in a paper by eight researchers, most of them at Google. Its title has become famous: "Attention Is All You Need." It was built for translation. The next year, OpenAI used the same design for its first generative pre-trained transformer, the model family that later gave ChatGPT its name.¹⁴

Attention is also why a chatbot can keep track of a long conversation. Everything it can see at once (your messages, its replies, any document you pasted) sits in what's called its context window, and attention lets each new word draw on any part of it. The bigger that window, the more there is to weigh, and the more computing each reply takes.²

The design turned out to work on more than words. Researchers cut an image into small patches and feed them to a transformer much like words in a sentence, and the same idea has been applied to sound. So when you ask a chatbot about a photo, a transformer is probably doing some of the work. Next time a translation gets a tricky word right, try changing one word next to it, and see whether the meaning flips.²