Be Wary of the Echo Chamber of Multi-year Surveys

If you conduct a periodic survey, such as a morale and engagement survey every six months, you can reasonably expect that what people read in a report (assuming you publish the results) will influence their answers the next round. Perhaps only slightly, but respondents may recall or be influenced by their latent memory when they answer the next iteration of the survey.

This expectation presumes that you maintain the same wording of the question and the same instructions for how to answer. If you materially change the question or guidance in the next survey, you lose precision regarding trend data, but you might lessen the echo chamber effect. A negatively phrased question one year followed be the inverse positively phrased the next year would mitigation the carry-over effect.

Consider the annual Norton Fulbright surveys of litigation data. Each year the firm polls hundreds of law departments in the United States and the United Kingdom.

If the responding group year after year remains significantly similar might not the results reflect some degree of self-reference (the echo chamber effect as they read, remember, and reinforce a self-view)? If one year’s results show a decline in trust in international arbitrations, to imagine one possible example, might the respondents read that finding and be influenced during the following year’s survey in their scores, beliefs, and interpretations because of what they vaguely remember from the report or publicity the previous year? They might not recall the source of their now-revised belief system. But they hear the echo of the survey’s prior findings and answer the repeated questions to some degree influenced by those whisperings.

The point raised here ranges more broadly. We could surmise a similar reverberation and influence from any repeated or annual survey. The echo chamber should have little effect on quantities that are counted, such as budget figures or staffing numbers. Its risk rises substantially on qualitative assessments and attitudes, such as how respondents regard the perceived effectiveness of alternative billing arrangements. When respondents have heard collective assessments one year, the bandwagon effect the next year may reinforce beliefs and thus influence answers in the same direction. Stated differently, the echo-chamber effect manifests a form of delayed peer pressure and group think.

Survey afficionados have a term for such a warping effect, which is a form of artefact or bias. The findings from last year’s survey lurk in the recesses of a respondent’s mind and influence how they answer this year. It could be they are stronger in their beliefs and give higher ratings to a question; or they might moderate a view a bit because of what they have absorbed from past surveys. This is to say that respondents should alter their views because of experience, including exposure to prior survey results, but that a self-reinforcing cycle might set in over the years for respondents who consistently answer a survey that asks the same attitudinal questions.