Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data
Source: https://aclanthology.org/2020.acl-main.463/ ↗
Full text: author page ↗
This is Searle's Chinese Room rebuilt for the technology in your stack.
Bender and Koller distinguish form (the observable signal of language) from meaning (the relation between that signal and communicative intent, grounded in a shared world) and argue that a system trained only on form cannot, in principle, learn meaning.
Their octopus thought experiment makes it stick: a creature that perfectly predicts the statistical form of two people's telegraph messages can impersonate them convincingly, yet has nothing to offer the moment one of them genuinely needs help in the world it has never touched.
The lesson for anyone building on language models is exact and uncomfortable: fluency is not comprehension, and "the model said so" is a fact about text, not about reality.
When you treat a model as a stand-in for a user, you are sampling the distribution of what has been written — form — not gathering grounded evidence.
Freely available from the ACL Anthology.
Central argument
Bender and Koller argue that a system trained purely on linguistic form — however much of it — cannot in principle learn meaning, because meaning is the relation between form and communicative intent that is grounded in a shared world the system never has access to. Their octopus thought experiment makes the claim vivid: a perfect predictor of message form can impersonate a speaker yet fail utterly the moment genuine, world-referring help is required. The paper is a call for conceptual hygiene, warning the NLP field against describing models that manipulate form as if they 'understand' or capture 'meaning.'
Critique
The paper's strong in-principle claim has been vigorously contested: some argue that form, at sufficient scale and with feedback, grounding, or multimodality, can bootstrap something functionally like meaning, and that the octopus's failure is a matter of degree rather than kind. Later work on grounded and tool-using models complicates the clean form/meaning binary. But even critics concede the paper's central discipline — refusing to conflate fluency with understanding — is exactly the hygiene the field lacked.
Why it matters for product
This is the single clearest contemporary statement of why 'the LLM told me' is not the same as knowing something about the world. For a product leader, the form/meaning distinction is directly operational: it explains why a model can generate a persuasive user persona or feature rationale that is ungrounded in any actual user, and why treating model output as research substitutes the distribution of written form for evidence about reality. Freely available from the ACL Anthology.