Transformer

Earlier language models read text one word at a time in order, which was slow to train and struggled with long-range connections. The transformer instead looks at all tokens in a sequence together and uses self-attention to decide which ones matter for each position. Because this computation parallelises well on GPUs, transformers can be trained on far larger datasets, which made today's large language models possible.

Why it matters: knowing that most AI products share this architecture explains common behaviour across tools: they work in tokens, have a fixed context window, and their cost grows with input length, because attention compares tokens with each other. Differences between products usually come from training data, model size, tuning and the software around the model rather than from the basic architecture.

Example: when a model translates "the bank by the river", attention helps it link "bank" to "river" and choose the riverbank meaning rather than the financial one.

Limits: standard attention becomes expensive as inputs grow, which is one reason long context costs more. Researchers continue to explore more efficient variants and alternative architectures.