SURVEY & QUESTIONNAIRE DESIGN: Collecting Primary Data to Answer Research Questions
SURVEY & QUESTIONNAIRE DESIGN: Collecting Primary Data to Answer Research Questions (55)
Jane Bourke, Ann Kirby, and Justin Doran
First stage
[list everything you already know about the problem that you are trying to solve.]
The research question is guided by the hypothesis. A research question is basically a hypothesis, formulated as a question.
A ‘good’ hypothesis must be: Adequate; Testable; Better than its rivals.
Next, identify your research objectives.
objectives are likely to lead to greater specificity than research or investigative questions.
SMART test: Specific: What do you hope to achieve? Measurable: Can you measure it? Achievable: Are the targets achievable? Realistic: What are your time constraints? Timely: Will you have the time to accomplish your objectives? Research objectives are statements of what you intend to find out when answering your question. Therefore identifying what you hope to measure with this research question and how you intend to measure it are two specific questions that need to be addressed.
The goal of standardisation is that each respondent be exposed to the same question experience, and that the recording of the answer be the same, too, so that any differences in the answers can be correctly interpreted as reflecting differences between respondents rather than differences in the process that produced the same answer.
Be mindful of how complicated the question is for the respondent to answer.
Avoid double-barrelled questions:
Take care with questions that have a prestige bias:
Avoid leading questions
Avoid direct leading questions – a closed question asking for agreement, pushing
Avoid negative questions. Sometimes the word ‘not’ in a question can confuse
Avoid assumptive questions -- Where did you go on holidays last year?
Only ask questions that respondents are willing to give correct and valid answers to
Beware of asking about information from proxies
Clarify what constitutes an adequate answer
Be mindful of your answer categories
Can the respondent comprehend the question that is being asked?
Is the respondent capable of answering the question that is being asked?
Is the respondent willing to answer the question that is being asked?
Confusing questions
Wordy questions
‘Single’ can simply mean ‘unmarried’, but respondents who are divorced or widowed may choose this option. The phrase ‘never married’ is a better alternative.
All questions about events and behaviour need a reference period – ‘last year’ or ‘in the last month
Non-factual Questions
Attitudes: A relatively stable opinion towards a person, object or activity, containing a cognitive element (perception and beliefs) and an emotional element (positive or negative feelings); Beliefs: Representing a person’s subjective knowledge about themselves and their world. Made up of
Direct experience, second-hand information, and inferences; Values: Enduring beliefs about imperative life goals that surpass specific situations:
Behavioural intentions: Attitudes toward behaviours tend to produce corresponding ‘behavioural intentions’. The degree to which these intentions are actually implemented is
Psychological trait: A basic, unchanging ability or skill – a long-lasting state of conditions such as intelligence, motives, emotions and values.
Psychological state: A mental condition, relatively constant but can fluctuate.
Open versus closed questions
Allow the researcher to obtain responses that were unanticipated;
Are time-consuming to code and tend to be unreliable (as coders may disagree which coding category to assign an answer to); Can be difficult to write:
Can be difficult to answer:
Closed questions: Allow the respondent to more reliably respond
Agree / disagree questions Are cognitively complex:
Are difficult to design; Are prone to acquiescence bias. A possible solution is to include a middle alternative,
A commonly-practised technique is to exclude the middle alternative but to pose a second question measuring the respondents’ intensity of feeling about their answer to the first.
Avoid using a “No opinion” or “Don’t know” option. Recent research shows that a “No opinion” option suggests to respondents that a great deal of knowledge is required
Satisfaction questions Satisfaction questions tend to be undiscriminating, in that they often give highly favourable results.
overall satisfaction measures are not a useful measure of performance as they give no specific guidance as to how the service can be improved.
Asking people to predict their response to a future or hypothetical situation should be done with considerable caution – particularly when respondents are likely to have limited direct experience on which to base their answers.
Fowler (1995) suggests that few of us are able to report validly on the reasons we do what we do. In fact, for many of our decisions / behaviours we give little or no thought to our decision-making.
Context effects Respondents’ answers to the current survey question can be affected by the previous survey questions.
