Reconsider Often-Skipped Questions

If you find during an online survey or after one closes that a good number of respondents didn’t answer a certain question, give extra thought to how to handle that question and its sparse answers. A question on a survey that returns lots of blanks suggests that it caused several types of problems. The respondents:

  1. didn’t understand the question or the instructions,
  2. didn’t want to take the time to research or construct an answer,
  3. felt comfortable offering their views on a sensitive topic,
  4. worried about anonymity,
  5. had become fatigued because the survey was too long,
  6. answered a conditional question that failed to navigate to the next question,
  7. deemed the question to have insignificant value to them and others,
  8. did not know enough to answer the question,
  9. recognized that the question delved into non-disclosable proprietary knowledge, or
  10. didn’t feel the question was applicable to them (“You’re asking me, an administrative assistant, about Continuing Legal Education??”).

Those reasons account for most of the situations where a question has numerous missing responses. If you notice the pattern promptly, for reasons 1 through 6 you might be able to clarify the question, state that approximate answers are acceptable, add a comment box, reinforce the assurances of anonymity, shorten the survey, move the question that suffers a paucity of answers earlier in the questionnaire, or fix the conditional logic. As to the final four issues, the right solution might be to drop the question that has problematic worth. Alternatively, you might include a selection for “Not applicable to me” or “Our firm deems this information to be confidential.” At least then you understand why the answer is blank.

If you are missing data for a question, imputation might be in order. This is a remedy only if you have enough data from elsewhere to fill in missing information with a defensible methodology. Another way to address the missing data is to ask for it or about it in the acknowledgement email. A few people might reconsider or will explain why they omitted an answer. You could even switch the question to make it required, but I would not favor that option. People shouldn’t be forced to give answers except to the most basic demographic questions, such as email address or country.

What should you do with sparse data other than imputation? My view is that if you troubled people for an answer, they deserve to have their input collated and reported, even if the number of responses to a question are paltry. You certainly have to add a caption to plots or state in the text the number of respondents to the question (“N = ”).