Build a Large Language Model (From Scratch)
Source: https://www.manning.com/books/build-a-large-language-model-from-scratch ↗
The fastest route from using models to understanding them.
Raschka builds a GPT from the ground up in PyTorch — tokenizer, embeddings, self-attention, transformer block, pretraining, finetuning — and each concept is earned by implementing the operation that produces it, never asserted in the abstract.
For anyone directing product and technology around AI, that difference is decisive: it replaces borrowed vocabulary with load-bearing understanding of what the labs actually do between a prompt and a token.
It is the spine of my month-long study track precisely because it teaches from the material, in the tradition of the people who learned computing by building it.
Central argument
A hands-on manual that constructs a GPT-2-scale language model end to end in PyTorch: working with text data (tokenization, byte-pair encoding, embeddings), coding self-attention and multi-head attention, assembling the transformer block and the full GPT architecture, pretraining on unlabeled text, and finetuning for classification and for following instructions. Its thesis is pedagogical: the only way to genuinely understand an LLM is to build one, deriving each concept from the concrete operation it performs.
Critique
By design it stops at a small, from-scratch model and does not cover the frontier machinery that separates a teaching model from a production system — large-scale RLHF/RLVR, mixture-of-experts, distributed training economics, serving and quantization at scale, or evaluation. It teaches the shared skeleton beautifully but leaves the reader to synthesize the last mile (the part where the labs actually compete) from elsewhere.
Why it matters for product
For a CTO/CPO, this is the single most efficient way to convert 'a model is an ocean of numbers' into precise mental models that make decisions tractable: what pretraining vs finetuning actually buy you, why context and tokens are the cost unit, where quantization trades fidelity for reach, and why capability is discovered by evaluation rather than specified. It grounds the build-vs-buy-vs-finetune conversation in how the thing really works.
- Man-Computer Symbiosis — J.C.R. Licklider
- The Rise of AI Search: Implications for Information Markets and Human Judgement at Scale — Sinan Aral, Haiwen Li & R. Zuo
- Beyond Human-Readable: Rethinking Software Engineering Conventions for the Agentic Development Era — Dmytro Ustynov
- AI Engineering — Chip Huyen
- Understanding Deep Learning — Simon J.D. Prince
- Responsive Web Design — Ethan Marcotte