Reproduce the Questions and Instructions in the Appendix

The body of a report should concentrate on succinctly explaining the survey’s findings, conclusions, and recommendations. It should not clutter the basic text with reproductions of the questionnaire elements. Often, however, on the top of a plot or table, the analyst will quote the question that produced the data shown. An excellent practice. The appendix to the report serves as the typical repository for the questionnaire.

As an exemplar, Greentarget and the Zeughauser Group lays out in the Appendix to their 2022 DEI survey not only the exact question asked but also a graphic that shows the distribution of answers.

Not only are the questions themselves rich with learning opportunities, but so are the instructions to the questions. Or, if you have inserted explanatory text before a cluster of questions, that too can help others be more precise and thoughtful. Even the order of questions contains gems of insight.

I have never seen screenshots of the online questionnaire from a survey. Those images would be highly instructive as they would reveal aesthetic and formatting techniques. Most sponsors close the link when the fielding period ends for a survey so you cannot access the originating questionnaire.

This admonition to reproduce the questions and instructions in the report aligns with the broader goals of reproducibility, credibility and intelligibility. Go beyond disgorgement of the data to a fulsome explanation of how the data emerged from specific questions.

The legal industry benefits if it sees full sets of questions and can borrow from their phrasing. Readers have more background to judge a report’s quality. Sponsors of later surveys can stand on the backs of giants. And other researchers can pull together data from multiple surveys when they know precisely what was asked and how. Finally, when others can see what you ask it helps them if they conduct a meta study or if they want to build on what you have asked.

The last point raises the idea of data amalgamation by different surveyors, a wonderful sounding idea that I have yet to see happen in real life. By that term I mean that sponsor combine their data to create even more value. For example, if one sponsor polled AMLAW 200 law firms regarding technology prowess and another polled the same group of firms regarding associate turnover and morale, it might be possible to find firms that took part in both surveys and amalgamate their data.

Concerns about loss of proprietary methods may lead sponsors to refusing such disclosure. Leave, without foundation in my opinion, that how they crafted inquiries and selections represents valuable intellectual property. I think not. Data might be preserved from view, but what led to it should be shared transparently.