Skip to content
A second look at the obvious answer
Think Twice TodayA second look at the obvious answer

Evidence

A number can become a fact by being cited often enough

Figures acquire authority through repetition and reformatting rather than through evidence, and a substantial share of the statistics in circulation cannot be traced to anything that measured them.

By Adrian Novak3 min read

Editorial note. Independent reporting and analysis. Nothing here is sponsored or paid for. How we work.

The figure everyone knows and nobody sourced

Every field has a few. A specific quantity, quoted confidently, appearing in presentations and articles and textbooks, always with the same value, usually with the same rounding, and rarely with a citation that leads anywhere. It is not that the number is necessarily wrong. It is that nobody using it could tell you how it was arrived at, and most have never wondered.

This site has already declined two of them in other articles: a widely repeated figure for how long it takes to regain concentration after an interruption, and a fixed quantity of practice supposedly required for mastery, which was a popular reformulation of research that said something considerably narrower. Both are quoted daily. Neither is supported in the form it circulates.

How the chain forms

The usual sequence has four steps and no villains. Someone produces an estimate, hedged appropriately, in a specific context. A second writer cites it, drops the hedge because the sentence reads better without it, and generalises the context slightly. A third cites the second, because the second is more readable and more recent. By the fourth, the citation points to a source that is itself citing something else, and the original conditions have vanished.

The most common single failure in this chain is the loss of the qualifier. “In this sample, under these conditions, roughly” becomes “studies show”, and the transformation happens in a single act of ordinary editing that nobody would identify as misrepresentation.

What makes a figure durable

Certain numbers survive and others do not, and the surviving ones share features. They are memorable, which usually means round or otherwise neat. They are actionable, in that somebody can build advice on them. They fit an existing belief, so nobody is motivated to check. And crucially they have no owner — no institution whose reputation is attached to them, and therefore nobody with a reason to correct the record.

Numbers with an owner behave differently. An official statistic gets revised, argued about and republished, because there is an organisation whose job it is to maintain it. Ownerless figures never get revised, because revision requires somebody to be responsible for the figure, and there is nobody.

Tracing one back

The exercise is instructive and takes about a quarter of an hour. Pick a figure you have used, find where you got it, then find where that source got it, and continue. The common outcomes are: a dead link, a citation to a secondary source that cites another secondary source, an original that says something narrower, or an original that turns out to be a passing remark rather than a measurement.

Occasionally the chain terminates in solid work and the figure is fine. That outcome is worth having too, because it converts a repeated number into a supported one and tells you what conditions it applies to. Either way the fifteen minutes buys more than reading another article that quotes it.

What to do with an untraceable figure

The answer is not to write around it with a vaguer statement that implies the same thing. If a claim rests on a quantity you cannot source, the honest options are to say the quantity is uncertain, to describe the mechanism without the number, or to drop the claim.

Describing the mechanism is usually the better piece of writing in any case. Readers can do something with an explanation of why interruptions are costly. A precise-sounding figure with no provenance gives them nothing except a statistic to repeat, which is how the chain extends by one more link.

The reader’s side of it

A useful habit is to treat any quoted quantity as carrying an implicit claim about its own history, and to notice when that history is absent. Phrases such as “research shows” and “studies have found”, with no study named, are the standard markers, and they appear most often precisely where the trail is weakest.

This does not require becoming the sort of person who demands a citation for everything. It requires holding sourced and unsourced numbers in different categories, and noticing when an argument depends on one of the latter. A great many do, and the dependence is invisible until you look for it.

There is a mild social cost to asking where a figure came from, and it is worth paying occasionally rather than routinely. Asked once, in the right place, the question is often enough to reveal that nobody in the room knows, which is a more useful outcome than any answer would have been.

Common questions

Does an untraceable number mean the claim is false?

No, it means the claim is unsupported in the form given. The underlying point may well be right, which is why describing the mechanism is usually better than either repeating the figure or discarding the idea.

Why do wrong figures survive so long?

Because they are memorable, useful and unowned. Nobody has a professional stake in maintaining them, so nothing triggers a revision, while their convenience gives many people a reason to keep quoting them.

How do I check a statistic quickly?

Follow it back one source at a time until you reach something that reports a measurement rather than a citation. Most chains terminate in under fifteen minutes, either in real work with conditions attached or in nothing at all.

Evidencecitationstatisticssourcesevidence
Adrian Novak
Staff writer, Think Twice Today

Adrian joined to cover biases, choices, risk and stayed for the awkward questions and would rather show the working than assert the conclusion.