Attention Is All You Need
The eight pages everything is built on.
Vaswani and colleagues proposed replacing recurrence and convolution with self-attention alone — queries, keys, values, multi-head attention, positional encodings — and showed it trained faster and translated better.
That modest 2017 claim became the substrate of GPT, BERT and every large language model since.
For anyone directing AI, it is the rare primary source worth reading whole: it turns 'attention' from a slogan into a specific, legible mechanism, and it is a small masterclass in how a narrow, well-argued idea can reorganize a field.
Central argument
The paper that introduced the Transformer, an architecture that dispenses with recurrence and convolution entirely and relies solely on self-attention to model dependencies between positions in a sequence. It presents scaled dot-product and multi-head attention, positional encodings, and an encoder–decoder design, and shows the model trains far faster (being highly parallelizable) while setting new state of the art in machine translation.
Critique
It is a 2017 machine-translation paper: it proposes the mechanism but not the scale, the pretraining recipes, or the decoder-only, generative direction that GPT would later take — those came in the years after. Read it as the origin point and the clean statement of the idea, not as a description of how today's LLMs are actually built and trained.
Why it matters for product
For a CTO/CPO, this is the single primary source worth reading in full: it is the architecture underneath every model you evaluate, buy or deploy, and it makes concrete what 'attention' actually means beyond the buzzword. It also models good engineering thinking — a narrow, well-tested claim that happened to reorganize an entire field.
- The Rise of AI Search: Implications for Information Markets and Human Judgement at Scale — Sinan Aral, Haiwen Li & R. Zuo
- Foundations of Large Language Models — Tong Xiao & Jingbo Zhu
- Hands-On Large Language Models — Jay Alammar & Maarten Grootendorst
- Build a Large Language Model (From Scratch) — Sebastian Raschka
- Understanding Deep Learning — Simon J.D. Prince
- Deep Learning with Python — François Chollet