Here we are considering a different notion than disqualifying a response that you decide is not legitimate. The idea is that a recipient’s answers to a survey – correct and complete though they may be – should not be included in the survey data set. The person competently answered the questions, but they are not a person whose views are sought. For example, ALM’s 2012 survey of alternative fee arrangements explained in the report that 218 law firms responded, of which 194 were qualified. The report does not explain why 224 (almost 10%) were deemed ineligible. They might not have been good faith responses or they might not have been target respondents.

Among the reasons for disqualifying participants, we have experienced the following:

Failed on qualifier: One requirement might be an answer to a key question. If they don’t check off software they use in a technology survey, they could be scotched. Here is an example. LexisNexis Interaction conducted a survey to quantify law firm RFP efforts. Whereas 359 people filled out the survey, only 213 of them quantified their monthly RFP activities, which was one of the first questions and the crucial gateway question to keep them in the data set.

Gateway question unanswered: A survey by Zapproved only wanted respondents who currently issued legal hold notices. They explained in their report that “While more than 800 individuals took the survey, the sample was further screened for those who currently issued legal hold notices,” which dropped the final set to 421 respondents. It would have been useful for readers to have included in a footnote the question(s) that led to the screening decisions.

Wrong title: If not a Chief Operating Officer or equivalent position, their views should not be included in a survey of COOs. No Deputy Chief Operating Officers or Chief Financial Officers who have no broader administrative remit.

Unfinished survey: You might disqualify those who don’t complete the survey. In one survey, of the 711 total respondents who started the survey, a total of 689 qualified with a complete. In my experience, quite a few respondents leave one or more questions blank, but that alone does not disqualify them. Just because the missing answers come at the end of the survey doesn’t make them more significant.

Too late for the deadline: It saddens me to welcome a respondent, but not be able to include their data in the report because they came to the party too late. My usual response is to send them the report anyway, as I hope they will appreciate it, spread the word, and perhaps take part in another survey later.