The art of survey design

Survey design is all about asking the right questions. In a classic study, Tversky and Kahneman (1981) asked their subjects the following question.

Imagine that the U.S. is preparing for the outbreak of an unusual Asian disease, which is expected to kill 600 people. Two alternative programs to combat the disease have been proposed. Assume that the exact scientific estimate of the consequences of the programs are as follows:

[Problem 1] If Program A is adopted, 200 people will be saved. If Program B is adopted, there is 1/3 probability that 600 people will be saved, and 2/3 probability that no people will be saved. [28 percent] Which of the two programs would you favor?

[Problem 2]  If Program C is adopted 400 people will die. If Program D is adopted there is 1/3 probability that nobody will die, and 2/3 probability that 600 people will die. Which of the two programs would you favor? (Tversky & Kahneman, 1981, p. 453)

The choice of their subjects was: program A (72%) ↔ program C (22%). Despite the fact that the options A and C (or B and D) mean exactly the same thing (200 people saved implicates 400 people died), the responses are almost opposite. This is, of course, the result of the ‘slightly’ different wording of the two questions. This and other examples resulted in the very influential, Nobel prize, prospect theory. You can read the now-classic article on-line, see references.

A more light-footed example comes from Bradburn, Sudman and Wansink (2004).

 “Two priests, a Dominican and a Jesuit, are discussing whether it is a sin to smoke and pray at the same time. After failing to reach a conclusion, each goes off to consult his respective superior. The next week they meet again. The Dominican says, “Well, what did your superior say?” The Jesuit responds, “He said it was all right.” “That’s funny,” the Dominican replies. “My superior said it was a sin.” The Jesuit says, “What did you ask him?” The Dominican replies, “I asked him if it was all right to smoke while praying.” “Oh,” says the Jesuit. “I asked my superior if it was all right to pray while smoking.” (Bradburn et al., 2004, p. 4)

There are several good resources for questionnaire design. Most of them are very practical with lots of tips and tricks. The classic text is without doubt “Asking questions” of Bradburn et al. (2004; revised edition). It is both very practical and profound and covers all aspects of questionnaire design. Each chapter ends with a checklist. You can read the combined checklists here. Of course, you should also consult the entire text and the countless examples from the book (see references for a link).

There is also a lot of empirical data concerning the various aspects of questionnaire design. An authoritative resource is the Handbook of Survey Research. You can read online the chapter  “Questionnaire design” of Krosnick and Presser(2010).

I give a short and biased preview of the chapter. Respondents execute four steps when answering a question and respondent errors should be resolved according to these cognitive factors. First, they must interpret the question and deduce its intent. Next, they must search their memories for relevant information, and then integrate whatever information comes to mind into a single judgment. Finally, they must translate the judgment into a response, by selecting one of the alternatives offered by the question.

• Open versus Closed Questions. The majority of survey questions are closed-form, mostly because of practical considerations. Open questions however can add richness to a survey and should in any case be used for the measurement of quantities (money spent, hours studied, …). Providing closed categories for quantitative questions (e.g. less than 5, between …) can bias respondent answers. Addition of a category “Others” is generally not effective.
• Number of Points on Rating Scales. Likert scales most often uses 5 points; Osgood’s semantic differential uses 7 points. There appears to be no standard for the number of points on rating scales. Reliability however is lower for scales with only two or three points compared to those with more points, but the gain in reliability levels off after about 7 points. Adding midpoints to rating scales improved the reliability and validity of ratings. A 7-points scale appears therefore optimal in many instances.
• Labeling of Rating Scale Points. Various studies suggest that reliability is higher when all points are labeled with words (not numbers) than when only some are (e.g. the extremes). However, people may sometimes “agree” regardless of the question being asked. For example, in one study 52% of people agreed with an assertion, whereas only 42% of people disagreed with the opposite assertion. Avoiding “agree/disagree”, “true/false”, and “yes/no” questions all together is recommended.
• The Order of Response Alternatives. Studies of response order effects seem to offer a confusing pattern of results, probably because of the failure to distinguish between presentation mode. A primacy effect (first items are selected) is observed in visual presentation, a recency effect (last items are selected) in oral presentation. Counterbalancing the order is only possible for a very limited (2-3) number of categories. Counterbalancing the items of a rating scale is not appropriate, except for the two primary sequences.
• Treatment of No-Opinion. Many surveys offer a “No opinion” or “Don’t know” option to ensure that respondent won’t guess. There is, however, considerable evidence that the “Don’t know” filters do not improve measurement. For example, when people are asked directly why they say “don’t know”, they rarely mention lacking information. Instead they most often cite other reasons such as ambivalence. Don’t know responses are also more likely when questions appear later in a questionnaire, at which point motivation to optimize is presumably waning.
• Social Desirability Response Bias. Misreporting (over- and under-reporting) in surveys is demonstrated in numerous cases: discrepancy of the answers with the official records, effect of bogus pipeline technique (researcher’s claim to know the answer changes the respondent answers), interviewer effects (for example, African-Americans report more favorable attitudes toward white people when their interviewer is white than when the interviewer is African-American and vice versa), effect of anonymity, … Several techniques are proposed to reduce social desirability; see next section for a fun attempt. The effect of anonymity is discussed in a separate post.
• Recall Error. This kind of error is mostly caused by mis-comprehension and the frailties of memory. Both types of errors can be reduced by by simplifying the task (for example, shortening the recall length; last month instead of last year) or by assisting the respondent in carrying it out (e.g. decomposing the question into multiple items).
• Question Order. Serial order can operate in at least three ways: by affecting motivation, promoting learning, and producing fatigue. This implicates that a questionnaire’s initial items should usually bear a strong connection to the topic and purpose that were described in the survey introduction, engage respondent interest, and impose minimal respondent burden. Learning, for example, occurs very fast in follow-up questions (most of the time following a “yes”-respons). Because these responses increase the questionnaire duration, respondents may falsely answer later screening items in order to avoid the contingent questions.

A rather funny attempt to ask embarrassing questions in a non-embarrassing way is described by Barton (1958). He tries to get an answer to the question Did you kill your wife?  If you like to see the Kinsey-technique in action, please take a look at the following trailer of the Kinsey-movie (2004).

REFERENCES

The post The art of survey design appeared first on Cogpsy.info.