Computer Power and Human Reason: From Judgment to Calculation
Source: https://archive.org/details/computerpowerhum0000weiz_v0i3 ↗
Full text: Internet Archive ↗
The man who built the first convincing chatbot spent the rest of his life warning us not to be fooled by it.
Weizenbaum's ELIZA (1966) faked a therapist with a few pattern-matching rules, and he was horrified to watch people confide in it and insist it understood them.
This book is the result: a sustained argument that computers calculate while humans judge, and that the danger is not machine cleverness but our own eagerness to project a mind onto mechanism — the "ELIZA effect." His conclusion is bracing and still contested: some decisions should be withheld from machines not because they cannot perform them but because delegating them is an abdication of responsibility.
For anyone tempted to treat a model's fluent reply as evidence, or a synthetic user as a stand-in for a real one, this is the indispensable and prophetic text — written by the one person who had the most reason to believe otherwise.
Central argument
Weizenbaum draws a hard line between what computers can do (calculation: the formal manipulation of symbols) and what they should be trusted to do (judgment: choices that require wisdom, values, and the whole of a human life). Drawing on his experience creating ELIZA and watching users attribute understanding and empathy to a program he knew to be empty, he argues that the real hazard is human: our readiness to see a mind where there is only mechanism. He insists that some domains — those involving interpersonal respect, care, and moral judgment — should be withheld from machine decision on principle, not because machines fail at them but because delegating them corrodes our sense of responsibility.
Critique
Written in 1976, the book's specific technical horizon is dated, and Weizenbaum sometimes verges on the mystical about an irreducible human essence that later cognitive science would contest. Critics note that his 'ought not' arguments smuggle in value commitments he treats as self-evident, and that the calculation/judgment line is blurrier than he allows. But the core diagnosis — that the seductiveness of fluent output is a human vulnerability, and that competence is not the same as fitness-to-decide — has aged extraordinarily well.
Why it matters for product
This is the foundational text for the governance question every AI-era product team faces: not 'can the model do it?' but 'should this decision be made by a system that calculates rather than judges?' Weizenbaum's account of the ELIZA effect explains, decades in advance, why stakeholders over-trust plausible model output and why a synthetic interview feels like contact with a user. It gives product leaders a principled vocabulary for drawing lines that competence benchmarks alone will never draw for them.