ABENAN
A questionnaire may look attractive and be easy to share, but one badly worded question can change the meaning of the answers. The problem is that a researcher cannot fix a question after the data have been collected, which is why review and piloting matter more than adding more participants.
1. The double-barreled question
Wrong wording: “The service is fast and easy to use.”
A respondent may find it fast but difficult, and will not know which option to choose.
Correction:
The service completes requests quickly.
The service is easy to use.
A simple rule: if you find “and” joining two qualities that could differ, review the question.
2. The leading question
Wrong wording: “To what extent do you agree that the excellent system improved the quality of work?”
Describing the system as “excellent” assumes the result in advance and encourages agreement.
Correction: “What effect has using the system had on the quality of work?”, with options from “reduced it a lot” to “improved it a lot”, or wording that suits the nature of the scale.
3. The emotionally loaded question
Wrong wording: “Do you support wasting resources on a little-used service?”
The words “wasting” and “little-used” carry a judgment that steers the answer.
Correction: “Do you support continuing to fund the service at its current level?” The reasons for the opinion can then be measured in a separate question.
4. Vague words
Wrong wording: “Do you use the platform regularly?”
What “regularly” means differs from person to person.
Correction: “In the last 30 days, how many times did you use the platform?”, with clear numeric categories.
5. No time frame
Wrong wording: “How many times have you faced a technical problem?”
Does this mean since you started using it, or this week?
Correction: “In the last three months, how many times did you face a technical problem while using the platform?”
6. The double negative
Wrong wording: “I do not think the instructions are not clear.”
The respondent has to untangle the negatives before answering, and may choose the opposite of what they think.
Correction: “The instructions are clear” or “The instructions are not clear”, paying attention to the direction of the coding.
7. Specialist terms
Wrong wording: “Was the system’s interoperability satisfactory?”
The term may be understood only by specialists.
Correction: “Did the system exchange data with other systems without problems?”, if that is what is actually meant.
AAPOR recommends using words and concepts the target audience understands, because participants differ in their knowledge and in their ability to understand terms.
8. Overlapping answer options
Wrong age categories: 18–25, 25–35, 35–45.
Where does someone aged 25 or 35 go?
Correction: 18–24, 25–34, 35–44, or collect age directly as a number if that is appropriate.
9. Options that are not exhaustive
If you ask how people access a service and list only mail and phone, an app user may not find a suitable option. Add the expected options after some initial research, and use “Other” when needed, but do not use it as a substitute for thinking through the basic categories.
10. An unbalanced scale
Wrong wording: excellent, very good, good, poor.
The scale has three positive levels and only one negative one.
Correction: excellent, good, average, poor, very poor, or a balanced satisfaction scale that suits the aim.
11. Forcing an opinion the respondent does not have
You may ask about a service that not everyone has used. If you do not offer “I have not used the service” or use a filter question, some people will pick a random answer. Use skip logic: those who have not used the service skip the questions that rate it.
12. Assuming prior knowledge or behavior
Wrong wording: “Why did you not renew your subscription?”
The question assumes the person was subscribed and did not renew.
Correction: start with a filter question, “Have you ever subscribed?”, then “Did you renew your subscription?”, and only then show the question about the reason to the right group.
13. The long, complicated question
The more parenthetical phrases and conditions a question contains, the more likely the respondent is to forget how it began. Shorten it, or split the scenario into a short introduction and a direct question. Do not try to explain the whole study inside every item.
14. Asking people to recall a long period
Asking a participant how many times a daily event happened over a year may produce a weak estimate. Choose a period they can remember, or use actual records if they are available. The period depends on the event; rare events may allow a longer range than daily behavior.
15. Mixing fact and opinion
“Do you think you used the platform five times?” combines an opinion and a count. It is better to separate the two:
How many times did you use the platform in the last month?
How do you rate your overall experience?
16. Question order effects
An earlier question can affect a later answer. If you first ask about all the problems and then ask about overall satisfaction, the respondent may become more negative. Order effects cannot always be removed, but they can be planned for: put general questions before detailed ones in many cases, and test different versions in sensitive studies.
17. Too many similar questions
Repeating an idea in near-identical wordings tires the respondent and may push them to choose the same pattern without reading. Every item should add a clear aspect to the dimension. After expert review and piloting, remove the repetition instead of keeping it to increase the number of items.
18. A confusing reverse-worded item
Not every reverse-worded statement is useful. If it is complicated or contains a negative, it may measure attention to language instead of the intended concept. Use such items carefully, keep them simple, and review them in the pilot test.
A practical way to review each question
Run every item through these questions:
Is it linked to a clear objective or dimension?
Does it measure only one idea?
Will the audience understand it on first reading?
Does it contain a judgment or steer the answer?
Does it need a time frame?
Are the answer options exhaustive, balanced, and non-overlapping?
Can the respondent know and remember the answer?
How will you code and analyze the answer?
Why is a pilot test necessary?
Read the question aloud, then ask people similar to your sample to explain in their own words what they understood. You may discover that a term you thought was clear has two meanings. The Pew Research Center stresses that a change of a single word or of question order can change the answers, which is why wording and order are reviewed carefully.
A final before-and-after example
Before: “Do you strongly agree that the useful and easy training greatly improved your skills over the past period?”
After:
In the month after the training, how much did your skill in using the system improve?
How easy was it to apply what you learned in the training?
How do you rate the usefulness of the training content for your work?
Each item now measures one aspect and has a clearer frame or meaning.
Before publishing your questionnaire widely, you can review ABENAN’s services for designing and reviewing questionnaires and preparing them for analysis.
Frequently asked questions
Is a long question always bad?
Not always; a question may need necessary context. The problem arises when it contains more than one idea, an unnecessary condition, or words that could be removed without losing the meaning. The aim is for the respondent to understand it on first reading.
Should every scale include a “neutral” option?
No. Whether to include it depends on the nature of the concept and the aim of the measurement. Sometimes it represents a real position, and sometimes it becomes a refuge for avoiding an answer. The decision should be methodological, while avoiding forcing people who have no opinion to choose a direction.
Is proofreading enough to make the questions correct?
Proofreading does not guarantee good measurement. A sentence can be grammatically correct and still be leading, double-barreled, or have unbalanced options. The review needed covers meaning, neutrality, recall, and the analysis plan.
Loading comments…