The wording of a question is part of the answer it produces

Same population, same issue, different number

It is an old and reliable finding in survey research that small changes to how a question is asked move the answers, sometimes substantially. The classic demonstrations are almost comic in their simplicity: asking whether something should be forbidden and asking whether it should be allowed do not produce mirror-image results, because people are more reluctant to forbid than they are unwilling to allow.

This is not a fringe result. It has been reproduced across topics and decades, and survey methodologists treat it as a design constraint rather than a curiosity. It is also distinct from the better-known argument about whether a measure captures the concept it claims to. Here the concept may be perfectly well chosen and the number still depends on the phrasing.

The parts of an instrument that move the answer

Four are worth knowing. The verb and its framing, as in the forbid and allow case. The response options offered, since people treat the middle of a scale as normal and the endpoints as extreme, which means a scale whose range is set high shifts everyone upward. Whether an explicit option to say “no opinion” is offered, which changes how many people give one. And the presence or absence of an explicit alternative in the question itself, since naming a specific alternative tends to raise support for it considerably.

None of these are mistakes by the researcher. Every question has to have some wording, some scale and some set of options, so there is no neutral instrument available to compare against. The number reported is a joint product of the population and the instrument, and the honest presentation says which instrument.

People agree more than they disagree

A separate and well-documented tendency is to agree with a statement as put, especially where the question is long, the topic unfamiliar or the respondent tired. This means a battery of items all phrased in the same direction will overstate whatever those items assert.

The standard fix is to reverse the wording of some items so that agreement means the opposite thing, and then to check whether the responses hold together. It is a good fix with a known cost: reversed items are harder to read and produce their own confusion, which is why the practice remains debated among people who build these instruments for a living.

The order of questions carries over

Questions asked earlier change the answers to questions asked later, because the earlier ones make certain considerations available and set a standard for comparison. Ask about a specific grievance and then about general satisfaction, and the general answer will be lower than if the order had been reversed.

It is worth separating this from the wave of social priming research that failed to replicate. Question-order effects in surveys are a much older and sturdier literature, demonstrated in large field samples with the real thing being measured, rather than in small laboratory studies with a distant outcome. The mechanisms proposed for them are ordinary — what is in mind, and what the respondent takes the question to be about — and they do not require any of the contested claims about subtle unconscious influence.

What this does not undermine

It does not make survey data worthless, and treating it that way is the lazy conclusion. Comparisons using the same instrument across time or across groups remain informative, because the instrument’s contribution is roughly constant and mostly cancels. A twenty-point change in the same question asked the same way is a real change.

What is unsafe is treating a single number from a single wording as the level of anything. The absolute figure is instrument-dependent; the differences and trends are much more robust, which is exactly the opposite of how such results are usually reported.

Reading one, or writing one

The first thing to look for is the actual question text, and its absence from a report is itself a finding. Then look at what was offered as alternatives, whether the scale was balanced, and what preceded it. Where two surveys disagree, the wording usually explains more of the gap than the sampling does.

If you are writing questions, the cheapest insurance is to ask the same thing in two different ways and compare. Agreement suggests the answer is about the topic. Disagreement tells you the instrument is contributing heavily, which is worth knowing before anybody quotes your number as a fact about the world.