When writing about online survey construction and analysis, I use the term “demographics” to describe the data collected about the person or organization that is responding. Not the insightful and new information that is aimed for, such as attitudes, rankings, percentage of agreement, and open text, but descriptive facts about the respondent. Demographic data include title, office location, education, years of experience, and gender, for individuals and revenue, employees, patents, headquarters country, and offices for organizations. They help you assess whether your survey data is representative of the population you have sampled from, and they help you convey inter-group differences in whatever you are studying. You can tabulate and visualize your survey findings according to demographic attributes. For example, you might survey your paralegals on software take-up, and then plot their technology prowess in boxplots by junior paralegal, paralegal, and senior paralegal.

If you have never created survey questions, you would be excused if you assume that putting together demographic questions is cut and dried. How many choices can there be for educational level? What can go wrong when you ask about the number of lawyers in a law department? Let me draw on several instances in a recent project to disabuse you that this category of questions is a slam-dunk.

   One question asked respondents for their level of education attained. Among the choices were “Four-year college degree” and “Business degree.” The question allowed multiple answers because some people earn more than one advanced degree. A problem arose when several people checked ¬both “Four-year college degree” and “Business degree.” Had they majored in business as an undergraduate or did they earn an undergraduate degree plus an MBA? Or is there a master’s degree in business that is short of an MBA? I ended up having to write those respondents to clarify their answer. More generally, if a hierarchy exists for your demographic selections, do you want all applicable answers or only the highest answer?

   The same survey asked for the title of the person. Painful experience had taught me not to permit an open-text response because the proliferation of titles bestowed on people renders standardization and analysis almost impossible. So, the question said, “Of the following list, which would be closest to your title?”. This pushed the answers into six or seven most common titles (Here is an example of the value of a subject matter expert (SME)), who can guide you on what to write for the selections. Even so, we added an “Other” selection and thereby learned a handful of unusual titles.

   Geographic scope of a person’s responsibilities or a corporation’s revenue introduces more pitfalls and trickery. Do you mix countries and regions? Are continents too broad? Someone will inevitably fall between (or across) the cracks, perhaps by being in charge of Mexico and Central America, which doesn’t slot into the selections of “North America” and “South America.”

Whenever you include an “Other” choice in your selections for a demographic question, you hope that few people need to resort to it. If many respondents choose “Other,” your list of selections failed to cover the likely terrain. Of course, you could leave that catchall category without an explanatory text box, but then you have a black hole – the infamous “Other” – that defies analysis or interpretation. But once you let its nose under the tent with a text box, you must decide how to classify the oddballs.

Careful thought, survey experience, and conscientious pretesting can reduce to a minimum the derangement of demographics, but the devil always creeps in. It usually means you must email or call the person to figure out what they meant, and keep track of their explanation, and correctly add it to your data set. This piecework is time-consuming, boring, and error prone. Or you can make unilateral decisions, defend them in the methodology section, and hope for agreement or sympathetic understanding.

Despite all your attention to detail and stress testing, real life often humbles you on demographic questions. If yours is a one-shot survey, you have no realistic chance to correct the mistake after all the responses have come back in. My advice is to look quickly yet carefully at the first few responses that show up and run through a checklist of the kinds of issues discussed here. It is far better to correct the problems as quickly as you can, maybe by revising selections, improving the instructions, or rewording the question. The inventiveness (or perversity?) and complexities of humans may still defeat you, despite all your efforts.