Asking Questions: The Definitive Guide to Questionnaire Design -- For Market Research, Political Polls, and Social and Health Questionnaires
Asking Questions: The Definitive Guide to Questionnaire Design - Norman M. Bradburn, Seymour Sudman, Brian Wansink
Questions must be precisely worded if responses to a survey are to be accurate;
We must depend on pretesting to weed out ambiguities and to help reformulate questions as clearly as possible-to ask about what we want to know, not something else.
Loaded Words Produce Loaded Results
Frugging (fundraising under the guise) surveys, are often primarily intended to raise funds rather than to collect survey information.
Sometimes questions are simply complex and difficult to understand.
Yet even when there are no deliberate efforts to bias the question, it is often difficult to write good questions because the words to describe the phenomenon being studied may be politically charged.
Respondents ... must be persuaded to participate in the interview, and their interest (or at least patience) must be maintained throughout.
Their only reward is some measure of psychic gratification-such as the opportunity to state their opinions or relate their experiences to a sympathetic and nonjudgmental listener, the chance to contribute to public or scientific knowledge, or even the positive feeling that they have helped the interviewer.
(1) a survey is a transaction between two people who are bound by special norms; (2) the interviewer offers no judgment of the respondents' replies and must keep them in strict confidence; (3) respondents have an equivalent obligation to answer each question truthfully and thoughtfully; and (4) in the survey it is difficult to ignore an inconvenient question or give an irrelevant answer.
In general, although respondents are motivated to be "good respondents" and to provide the information that is asked for, they are also motivated to be "good people." That is, they will try to represent themselves to the interviewer in a way that reflects well on them.
Techniques for helping respondents resolve this dilemma on the side of being good respondents include interviewer training in methods of establishing rapport with the respondent, putting respondents at their ease, and appearing to be nonjudgmental.
Investigators should try to avoid asking respondents for information they do not have.
The term informed consent implies that potential respondents should be given sufficient information about what they are actually being asked and how their responses will be used. The intent is for them to be able to judge whether unpleasant consequences will follow as a result of their disclosure.
The fact that the researcher has promised confidentiality to the respondents will not protect the researcher from having to produce the individual records if required by legal action.
First you will need to identify the concepts involved in the research question. Then you will formulate specific questions that, when combined and analyzed, will measure these key concepts. For example, if you are interested in the attitudes of potential voters toward a particular candidate, you will have to decide which attitudes are important for the topic at hand: attitudes about the particular positions the candidate holds, attitudes about the candidate's personality, or attitudes about the candidate's likability.
As a general rule, when constructing a questionnaire, you must continuously ask "Why am I asking this question?" and must, in each instance, be able to explain how the question is closely related to the research question that underlies the survey.
Given our biases toward more information, a game of "Wouldn't it be nice to know?" can quickly ensue, and soon there are many more questions than the budget can afford or than respondents can endure.
Every time you write a question, ask yourself "Why do I want to know this?" Answer it in terms of the way it will help you to answer your research question.
Bias refers to an estimate that is either more or less than the true value. Variability is measured by the susceptibility of measurements to differences in question wording.
The incorrect placement of events in a particular time period is called telescoping.
Forward telescoping typically results in overreporting of events; backward telescoping typically results in underreporting.
Another source of error arises from the deliberate overstating or understating.
Another source of error stems from the respondent's failure to understand the question in the way the researcher intended.
Finally, respondents may simply be ignorant [and because people don't like to appear ignorant, they may just make something up or guess or say what they think is wanted.]
The most direct and probably the most common questions asked of respondents relate to their behavior.
When asking a closed-ended question about behavior, make sure that all reasonable alternative answers are included.
Make the question as specific as possible.
The time period of the question should be related to the saliency of the topic. Periods of a year (or sometimes even longer) can be used for highly salient topics, such as purchase of a new house,
Periods of a month or less should be used for items with low saliency, such as purchases of clothing and minor household appliances. Periods that are too short, however, should be avoided,
For regular, frequent behavior, respondents will estimate the number of events by using the basic rate they have stored in memory.
Where detailed information on frequent, low-salience behavior is required, providing diaries will result in more accurate results than memory.
Use words that virtually all respondents will understand.
Do not assume that the shorter questions are necessarily better.
Respondents may not know what is meant by the word "regularly." y specifying "on a daily basis," the question removes or reduces the uncertainty.
A general finding is that as the number of experiences of an event increases above five, respondents are more likely to estimate than to count.
Vary question formats where possible, to make the interview more engaging for the respondent and also to decrease the chances of respondent anticipation.
One simple reason for making each question as specific as possible is to make the task easier for the respondent, which, in turn, will result in more accurate reports of behavior.
The "when" question should specify the time period by using actual dates instead of terms such as "last week" or "last month."
If general or global questions are used, they should be tested to determine what respondents think they mean.
The more important the event, the easier it is for the respondent to remember.
there appear to be three dimensions that distinguish between events that are more and less salient: (1) the unusualness of the event, (2) the economic and social costs or benefits of the event, and (3) the continuing consequences of the event.
Diaries have been used for frequent, nonsalient events that are difficult to recall accurately.
The general principle is simple: use words that everyone in the sample understands and that have only the meaning you intend. Writing questions that satisfy this principle is a difficult art that requires experience and judgment.
Explain the word first and then provide the word itself.
Slang and colloquialisms should normally be avoided, not because such words violate good usage but because many respondents will not know what the words mean.
When surveying unfamiliar groups, an initial group interview with a small (nonrandom) sample of that group may be helpful in indicating the types of words to use or avoid.
even more troublesome than an unknown word is a word that has multiple meanings in the context of the question being asked.
For socially desirable behavior, the extent of overstatement depends not only on the level of desirability and the wording of the question, but also on the proportion of the population who have not behaved in the socially desirable manner.
There is, a general tendency for respondents to avoid extreme answers and to prefer an answer in the middle of a list
Setting out rules for formulating questions about attitudes is more difficult than for behavioral questions because questions about attitudes have no "true" answer.
Attitudes exist only in a person's mind.
Context in which questions are asked has a greater impact on attitude measurement than on behavior questions because the meaning of the questions may be strongly influenced by the context in which they appear.
The terms opinion and attitude are not clearly differentiated from one another. In general, opinion is most often used to refer to views about a particular object such as a person or a policy, and attitude is more often used to refer to a bundle of opinions that are more or less coherent and are about some complex object.
People are less likely to believe something derogatory about something they like and are in favor of, and they do not usually act in support of things they disapprove of.
Similar (if not synonymous) terms that indicate a positive orientation toward an attitude object may have somewhat different connotations and yield different responses.
Strongly held attitudes are generally more resistant to effects of question wording than are weakly held attitudes.
The fundamental idea behind Likert scales is that an attitude can be thought of as a set of propositions about beliefs, evaluations, and actions held by individuals.
Another commonly used scale type is the Guttman scale. Items in a Guttman scale are ordered such that some items should be agreed to only by those who are low on the attitude and others should be agreed to only by those who are high on the attitude scale.
Various methods for combining responses can be used. The simplest is to count the number of "yes" and "no" answers as appropriate.
Unipolar items, when rephrased into what appear to be their opposites, often produce surprising results. A famous study by Rugg () showed that even such apparently opposite words as "allow" and "forbid" can produce dissimilar results.
"Do you think the United States should allow public speeches against democracy?" and "Do you think the United States should forbid public speeches against democracy?" When the question was one of allowing public speeches, percent of the respondents supported free speech; when the question was phrased that the United States should forbid free speech,
The potential biasing effect of the positioning of questions in a questionnaire has long been recognized.
The order of questions provides a context within which questions are answered.
When a general question and a more specific-related question are asked together, the general question is affected by its position, whereas the more specific question is not.
If the specific question triggers positive associations, it appears to increase positive responses to the general question. If the thoughts aroused by the specific question are negative, the effect appears to be negative. The specific question may narrow the interpretation to the meaning of the general question and have a corresponding effect on answers to the general question.
Unintentionally Activating Norms and Values that Cause Biases
Questions involving the same underlying value (reciprocity) are asked about objects with differing degrees of popularity. When the more popular item comes first, it appears to have the effect of heightening the value, so that it applies in the second and less powerful instance.
Use open-ended questions sparingly; they are primarily useful for developmental work, to explore a topic in depth, and to obtain quotable material. Closed-ended questions are more difficult to construct, but they are easier to analyze and generate less unwanted interviewer and coder variance.
Although not so common as behavioral questions, knowledge-related questions have many uses in surveys. They can be used to help explain political behavior, which is strongly impacted by one's level of knowledge.
Before asking attitude questions about issues or persons, ask knowledge questions to screen out respondents who lack sufficient information or to classify respondents by level of knowledge.
If yes-no questions are appropriate, ask several on the same topic to reduce the likelihood of successful guessing.
The line between knowledge and attitude or opinion questions is often blurred.
The question that asks respondents to guess about the proportion of welfare chiselers is really an attitude question in the guise of a knowledge question.
Customer ratings are becoming more common as organizations become more customer-oriented. In addition to controlled, formal surveying of target populations, most large companies now have continuous
Feedback mechanisms involving comment cards, toll-free hot lines, and on-line Web surveys.
A mechanism used to assess service across a wide number of units is "mystery shoppers," people who are evaluating the company by simply behaving like a shopper (or diner) and whose identity is not known to those they are evaluating. The primary job of mystery shoppers is to frequent the franchised restaurants to ensure that food quality, timing, cleanliness, and other standards are being met.
In many cases, customers are often in a better position to evaluate the quality of products and services offered than are managers or fellow employees.
This third-party performance rating can be critical for accurate product or service ratings in an industry.
When the term psychographics was introduced by Emanuel Demby in the s, it was generally defined as "the use of psychological, sociological, and anthropological factors, self-concept, and lifestyle to determine how the market is segmented by the propensity of groups within the market-and their reasons-to
The most obvious use of psychographic research is to draw portraits or profiles of target groups.
Psychographic questions are typically used to segment people by the way they think or behave.
Using psychographic questions involves using one or two hypothesized characteristics (or personality traits) to explain differences in choice or behavior. For instance, one study hypothesized that a trait called "venturesome" was related to the probability that "venturesome" people would try new and different products. Researchers found that people with this trait were more likely to choose a new and different flavor of toothpaste
List of Values (LOV), which is gaining favor among academics because it is in the public domain and relates closely to consumer behavior and to trends
For instance, a study attempting to understand (and deter) binge drinking interviewed bartenders to try to develop personality profiles of those most predisposed to binge drinking. Similarly, a study of men who spent a lot of money on business shoes used shoe shiners in airports to get a better idea as to how a "Cole-Haan" man differed from an "Allen-Edmunds" man (Wansink, c).
Conduct in-depth interviews or focus groups.
Although correlations between psychographic variables and preferences seldom get higher than .3 or.4 the same is true of the relationships between demographics and product preference. [0 is no correlation and 1 is absolute correlation.]
Ultimately, every questionnaire must be tested and refined under real-world conditions. Testing takes the form of pretest interviews and of soliciting peer feedback of draft questionnaires
Ask respondents if the questions were straightforward and whether the format made logical sense.
It is always useful for questionnaire designers to play the roles of respondents and answer their own questions.
One common mistake is to include modifying adjectives and adverbs that are somewhat unclear, such as usually, often, sometimes, occasionally, seldom, and rarely. These words have highly variable meanings, as do such words as many, most, numerous, a minority of, a substantial majority, a considerable number of, a large proportion of, a significant number of, and several.
And. The word and can signal that you might be combining two questions and asking them as one question.
Or. Similar to the word and, the word or is often associated with a double question or with a false dilemma.
If. The word if is often associated with confusing directions or with skip patterns.
Not. Avoid using not in your questions if you're having respondents answer "yes" or "no" to a question. Using the word not can lead to double negatives and confusion.
A good question is one that yields a truthful, accurate answer.
A Good Question Asks for Only One Answer on Only One Dimension
Each question should be about one topic. Do not include questions that require a single response when two would be more appropriate.
Asking a multiple choice question that does not accommodate all possible responses can confuse and frustrate the respondent,
A good question leaves no ambiguity in the mind of the respondent.
A Good Question Produces Variability in Response
When a question produces no variability in responses, we learn very little.
Writing a questionnaire is similar to writing anything else. Transitions between questions should be smooth. Questions should be grouped so that they are similar and easier to complete.
People generally look at the first few questions before deciding whether or not to complete the questionnaire.
Make respondents want to continue by putting interesting questions first.
