Prediction and Entropy of Printed English
Next-token prediction, by hand, in 1951.
Shannon measures the entropy of English by turning it into a guessing game — show a reader some text, have them predict the next letter, and count how well they do — and from that derives how predictable, and how redundant, the language is.
That is precisely the objective a modern LLM optimizes (predict the next token; minimize the surprise) and the metric it is judged by (perplexity is Shannon's entropy in disguise).
Where his 1948 paper founded information theory, this one points it straight at language — which is why it reads as the missing link between Shannon and the transformer.
A rare primary source you can read in one sitting and never see next-token prediction the same way again.
Central argument
Shannon measures the entropy — the average information per letter — of printed English, and does it experimentally: by having human subjects predict successive letters of a text and recording how often they succeed. From this he derives bounds on the entropy and the redundancy of English, and models the language with n-gram statistics. It is next-symbol prediction as a way to quantify a language's structure.
Critique
It is a short, information-theoretic study of English orthography from 1951 — it says nothing about learning, neural networks, or generation, and its estimates are of their time. Its value is conceptual and historical: it establishes the prediction-equals-understanding-of-structure idea that language modeling would later mechanize, not any method you would use directly.
Why it matters for product
For anyone learning how LLMs work, this is the origin story of the training objective itself. It makes concrete why 'just predict the next token' is not a trick but a measure of how much a model has captured the structure of language — and it grounds perplexity, the metric you will meet in every training run, in Shannon's entropy. The clearest one-sitting read for seeing that modern language modeling rests on a 1951 idea.
- A Mathematical Theory of Communication — Claude Shannon
- When Life Gives You AI, Will You Turn It Into A Market for Lemons? Understanding How Information Asymmetries About AI System Capabilities Affect Market Outcomes and Adoption — Alexander Erlei, Federico Maria Cau, Radoslav Georgiev, Sanjay Kumar & Kilian Bizer
- Single-Agent LLMs Outperform Multi-Agent Systems on Multi-Hop Reasoning Under Equal Thinking Token Budgets — Dat Tran & Douwe Kiela
- The Rise of AI Search: Implications for Information Markets and Human Judgement at Scale — Sinan Aral, Haiwen Li & R. Zuo
- The Compulsory Imaginary: AGI and Corporate Authority — Emilio Barkett
- Belief Updating and Delegation in Multi-Task Human–AI Interaction: Evidence from Controlled Simulations — Shreyan Biswas, Alexander Erlei & Ujwal Gadiraju