Qualify your conjectures

In relation to any experiment we may speak of this hypothesis as the "null hypothesis," and it should be noted that the null hypothesis is never proved or established, but is possibly disproved, in the course of experimentation. Every experiment may be said to exist only in order to give the facts a chance of disproving the null hypothesis.

— The Design of Experiments, R. A. Fisher

Fisher wrote this in 1935, and it is one of the strangest sentences in the history of method. Read it again slowly. An experiment exists in order to give the facts a chance. Not to give the researcher a chance. The facts are the agent; we are merely the ones who arrange the encounter.

The direction of the burden

The null hypothesis is the assumption that your change makes no difference. Stating it first is not a formality. It is what fixes the burden of proof, and it fixes it in the direction that costs you something.

Product organisations almost invariably do the opposite. The hypothesis is written as we believe that adding X will increase Y, and the entire apparatus is then pointed at confirmation. This is not dishonest, which is what makes it hard to correct. Everyone involved wants a true answer. They have simply arranged an inquiry that can only return one kind of answer loudly and the other kind quietly.

Fisher's asymmetry is the corrective. You never prove the effect. You fail to disprove the absence of one, and every so often the absence fails to survive contact with the data. It is a humbler epistemology than the one on most roadmap slides, and it is the only one that has produced durable knowledge at scale.

Popper generalised it into a theory of knowledge: a claim that cannot be refuted has told you nothing. Peter Medawar, who spent his life doing the work rather than theorising about it, added the practical corollary — the researcher's skill is choosing which refutable questions are currently affordable.

What qualifying actually involves

Qualifying a conjecture, before anything is built, comes down to answering two questions in writing.

The first: what result would count as a refutation? Not what result would disappoint us, which is a feeling, but what observation would make us stop. The value of writing it down is that most of the time you find you cannot. You discover you would have found a way to proceed under any plausible number. That discovery is worth the ten minutes it costs, because it tells you the metric was never load-bearing.

The second: what would it cost to obtain that result, and is the cost proportionate to the decision? Kohavi, Tang and Xu are careful about this in a way that most enthusiasm for testing is not. An underpowered test does not return a weak signal. It returns a number with full confidence attached and no relationship to reality, which is considerably worse than having no number, because a number cannot be argued with in a meeting.

James Flier's account of irreproducibility in bioscience is the sobering companion. A field with far more rigour than ours, and far higher stakes, still produces a large volume of published findings that do not survive replication. The method is not a guarantee. It is a discipline, and disciplines decay when nobody is watching.

The honest alternative

None of this is an argument that every initiative needs an experiment. Most do not, and pretending otherwise produces the theatre described in the session on experimentation: the word pronounced in capitals, the scaffolding underbuilt, the results interpreted by the person who proposed the change.

The honest alternative is to say that this one is a bet on judgment. Annie Duke's framing is useful precisely because it removes the pretence: you are wagering under uncertainty, your judgment is the instrument, and the way you improve the instrument is by recording what you expected before you find out. Not after. Afterwards, memory does the editing for you, silently and generously.

So: qualify your conjectures. Not all of them, and not to the same depth. But know which rung of the ladder each one is standing on, and know it before you commit the money.

Fisher gave the facts a chance. The least we can do is not arrange the room so they cannot speak.