Understanding the Affordances of Control in AI Reasoning for Human-AI Decision-Making
Source: https://www.semanticscholar.org/paper/6c94c6d875a588d112712379d11d8841618d6ffa ↗
The paper's central finding is unsettling in a precise way: giving users the ability to edit AI reasoning increases their sense of control but also increases over-reliance when the AI is wrong — an illusion of control that is worse, in practice, than no explanation at all.
Read-only chain-of-thought explanations, the current industry default, perform worst of all by inducing agreement without engagement.
This inverts the standard explainable-AI design rationale, which assumes that more transparency reliably improves human judgment.
For product directors building AI-assisted decision tools, the empirical result — not just the intuition — that transparency mechanisms can undermine the agency they appear to restore is a structurally important finding.
The work belongs alongside Simon's bounded rationality canon and the behavioral economics thread in the library, grounding abstract concerns about human-AI collaboration in a controlled experiment with a clear psychological mechanism.
Central argument
Moon, Huang, and Xiao find that different affordances of AI reasoning transparency produce meaningfully different decision-making outcomes — and not in the direction standard XAI design assumes. Editable chain-of-thought reasoning increases users' subjective sense of control but simultaneously increases over-reliance when the AI is wrong, creating an illusion of control that is empirically more dangerous than no explanation. Read-only chain-of-thought, the current industry default, performs worst for genuine engagement: it induces agreement without critical processing. The paper's core thesis is that the form transparency takes — not transparency itself — determines whether humans exercise or surrender judgment.
Critique
The controlled experiment necessarily abstracts away the conditions under which real AI-assisted decisions are made: time pressure, professional accountability, domain expertise, and organizational incentives to defer to AI recommendations. Over-reliance measured in a lab task may not scale linearly to high-stakes professional contexts where users have skin in the game and domain knowledge to contest AI outputs. The finding that editability worsens outcomes when the AI is wrong also sidesteps the question of calibration — if users could accurately identify when the AI is wrong, the intervention might perform differently, meaning the result may be as much about AI reliability signaling as about the affordance design itself.
Why it matters for product
For a CPO designing AI-assisted workflows — prioritization tools, risk scoring, recommendation engines — this paper reframes the design question from 'how much transparency should we expose?' to 'what form of transparency produces the decision behavior we actually want?' The finding that read-only explanations are the worst performer is a direct challenge to any product team shipping chain-of-thought reasoning as a trust-building feature without instrumenting how it changes downstream user behavior. It also raises a governance question: if editability creates an illusion of control rather than genuine oversight, product leaders need explicit criteria for what counts as meaningful human-in-the-loop design, not just a UI affordance that satisfies a compliance checkbox.