the transformer

The word reaches wider than the systems most people meet it in. Every current large language model is built on the transformer, and so was a DeepMind network that played Atari, captioned images and stacked blocks with a robot arm. One architecture, several kinds of system. The language models are the famous part rather than the whole of it, and the breadth is the first thing to get straight, because the word gets used as a synonym for a chatbot.

The source is one paper. Attention Is All You Need, Vaswani and seven co-authors at Google, eight authors in total, arXiv 1706.03762, v1 12 June 2017. It is the paper the T in GPT points at.

So "transformer" names a shape of network, not a product and not a kind of task. Gato is the clearest case of that: a transformer that was not doing language. The same shape also carries more than one way of training it, which is where the word most often gets misused.

The boundary that matters is between the architecture and the way a model is trained, because those are separate choices and get confused constantly. Masked pre-training, hiding words in a text and making the model guess them, is the method of BERT, from Devlin, Chang, Lee and Toutanova at Google, arXiv 1810.04805, 11 October 2018, as against GPT's next-token prediction. Same era, different kind of fact: one names the shape of the network, the other names what you do to it. The p(doom) song, annotated dates the architecture to 12 June 2017 and calls masked pre-training the one piece of machine learning in the song that is not what the chatbots use.