When absence of evidence is evidence of absence, and when it is not

The slogan is half right

The familiar line says that absence of evidence isn’t evidence of absence, and it’s used to defend claims that nothing supports. As a general principle it’s wrong, and the correct version is more useful: failing to find something is evidence against it, in proportion to how likely you would have been to find it if it were there.

That proportion is the whole content. Search a small room thoroughly for a large object and not finding it is close to decisive. Search a coastline briefly for a small one and not finding it means almost nothing. The same negative outcome carries a completely different weight in the two cases, and nothing about the outcome itself tells you which case you’re in.

Sensitivity of the search is the variable

The question to ask of any negative result is what would have had to be true for the search to have detected the thing. That covers the instrument, the sample size, the duration, the place looked, and whether the thing would have left a trace at all. Each of those can quietly reduce the detection probability to something near zero, at which point the negative finding is uninformative rather than reassuring.

This is why a study reporting no effect is not the same as a study reporting no effect worth having. A small trial can fail to detect an important effect simply by being too small, and the correct description of that result is that it was inconclusive. Treating it as a demonstration of absence is one of the most common misreadings in the interpretation of research, and it happens in both directions depending on which conclusion the reader preferred.

The strong version of a negative result

A negative finding becomes powerful when the search was well designed to find the thing. A large, preregistered study with adequate power, a sensitive measure and a clearly specified prediction, returning nothing, is a substantial piece of evidence — often more informative than the positive result it failed to reproduce.

The most convincing form states in advance what size of effect it could have detected, so that the negative result comes with a bound: whatever is there, it is smaller than this. That converts an absence into a measurement, which is a far more useful object than a bare report that nothing turned up.

Where the reasoning is abused

The slogan is most often deployed to protect a claim from the failure of every attempt to support it. After enough well-designed searches have come back empty, insisting that absence proves nothing is no longer a point about inference; it is a refusal to let the evidence count in the only direction it can move.

A related move is to explain each failed search individually — the method was unsuitable, the conditions were wrong, the wrong people looked. Any single such explanation may be entirely reasonable. A sequence of them, each invoked after the fact, describes a claim that has been made unfalsifiable by accumulation, and the pattern is more informative than any of the individual explanations.

The everyday versions

The same reasoning governs ordinary judgements more than people notice. No complaints does not mean no problem unless complaining was easy and likely. No symptoms does not mean no condition unless the condition produces symptoms early. No record of a decision does not mean it was never made unless somebody was reliably writing decisions down.

In each case the question is identical: if the thing were true, what would I be seeing, and would I have seen it? Where the answer is that the evidence would probably have been invisible anyway, silence tells you nothing, and building confidence on it is building on the absence of a search rather than on the results of one.

Making your own negatives worth something

If you want your failures to find something to count, decide beforehand where you would look and what would constitute an adequate look. A search designed after a disappointing result tends to be shaped, without any intent, by what has already been ruled out, which makes its negative outcome much less informative than it appears.

And write down what the search could have detected, in whatever crude terms are available. A note saying that you checked six months of records and would have caught anything happening more than a handful of times is a far stronger statement than a note saying you looked and found nothing. The first bounds the possibility; the second only reports an event.