My heart is committed to open-source software and code and my abiding faith rests with transparency in research: tell people what you did and how you did it and share as much data as you can so that your investigation is reproducible. Theoretically, others can retrace your steps and recreate your analytic conclusions. Openness and sharing help us all make faster progress in understanding our world.

That credo espoused, however, I must admit that many surveys in the legal industry have sponsors who do not wish to disclose much about their effort. Charitably, they don’t think readers of their reports care about methodology and how the sausage was made. More plausibly, they do not want readers to be able to question their findings and take them with chunks of salt. Most realistically, they want to give nothing away to their competitors.

What pieces of a survey project do sponsors regard as proprietary? What do they want to keep cloaked in secrecy? Here are several sensitive topics.

How they obtained participants: A sponsor shies away from admitting that they invited only customers and a somewhat wider range of prospects or contacts. That group is hardly a random set of the entire population: users of XYZ case management software represent a non-representative, partial slice of all law departments, for example. Moreover, users or clients may not answer as honestly as would respondents who have no ongoing business relations with the sponsor. Nor would a sponsor like to point out that they made many pleading calls to targets to persuade them to respond nor let slip what incentives they might have offered. Finally, a sponsor might not want to divulge that they purchased a list of email addresses and spammed emailed that group.

What software they used: To me, it is not worthy of keeping secret that you used a particular hosting software, or that you created the graphics with seaborn in Python or Tableau. If you use Visme, Prezi, or PowerPoint for the report, who can seize an advantage from knowing that?

Which analytic steps did they try: Sponsors probably do not wish to share the analyses that they carried out that led nowhere or to unfavorable findings. The suppressed truth might be that users of the sponsor’s software have higher total legal spending as a percent of revenue! Can’t let that out! What if they tried multiple linear regression and no predictor variable was statistically significant? What if they failed to address missing numbers or imputed them with curious values? Perhaps they published averages of averages instead of averages of individual numbers.

How did they phrase questions: In the view of some sponsors, the precise working of the questions and their instructions needs to be held close to the chest. No one can steal your brilliant phrasing nor quibble about ambiguities or leading questions. The less said, the better.

Where did they obtain augmented data: Perhaps a sponsor enriched the data collected by survey with data from external sources, e.g., populations of cities or market capitalization. The sponsor might deem it confidential data in terms of how they obtained it and what they did with it.

Planned developments: A previous section discussed the sponsor’s frank assessment of a survey project after it finishes, and a look ahead to what the sponsor would like to tackle in the next step. But cautious sponsors, wary of giving away too much to critics and competitors, might shield that speculation from the public eye. In the fight for market share, don’t telegraph your punches!

Project team members: No sponsor introduces the team that worked on the survey project and what their roles were. With collaborators silence follows on their respective contributions. It’s as if a disembodied robot cranked out the survey and report.

How much time and money was invested: Above all, don’t say how much you spent. Resistance to that disclosure has internal rationale: partners might object if they knew that someone authorized $24,000 for a survey that helps one practice group. Where is the return on investment? The external rationale is that our business is nobody’s business, and we don’t want invidious comparisons or revelations about our marketing efforts. Besides, you can’t take a lawyer’s hourly billing rate, multiply it by the hours they work on a survey, and add that amount into the investment total.

What methodology decisions did they make: This umbrella term – methodology – means for me almost everything that the sponsor did to produce the survey report. Aside from what is mentioned above, it could be the number of email invitations and reminders sent; the qualifying questions used; the efforts to ask about missing or misleading responses, or an adjustment to create trend data.