Many survey sponsors ask respondents to rank a set of selections, such as to allocate 1 to 7 for seven selections. The average of each selection’s rankings becomes findings in the report, often visualized as a bar or column chart (a bubble chart offers an alternative perspective). The charts usually display the results in declining average rank order. But often whoever prepares the report fails to focus on whether the gaps between the averages are roughly equal.

Here is an instance from real life. Respondents to one of ALM’s surveys ranked seven possible internal obstacles that law firms face when implementing alternative pricing strategies. The leading reason was “Law firms are more comfortable with the billable hour” (2.2 average ranking where 1 is most important). Next was “Absent better metrics, it is difficult to determine alternative values” (2.9), followed by “Firms have insufficient experience defining or managing work on an alternative basis” (3). After that, “There is not sufficient billing history or pricing methodology to know how to bill AFAs” (3.6), “Other” (4), “Alternative fee arrangements are too risky for the firm’s overall revenue” (4.3) and lastly “Partners object or refuse to cooperate” (6).

The gaps in the average ratings highlight how much the respondents thought the first reason dominated.^[Medians of the values could also serve, but not as well because they would be whole numbers.] The gap between that leading reason’s average and the next lower average is 33% of the lower average. “Absent metrics” and “insufficient experience” are in a virtual dead heat (2.9 and 3) so they were perceived to be nearly equivalent obstacles. Then a 20% gap opens between the third ranked reason (insufficient experience”) and the fourth (“insufficient billing history”), which is closer to the fifth (“Other” at 4). Partners’ objections were off-the-chart low.

I was struck by the uneven gaps between some of the rankings. The ALM report did not pick up on this observation or what might be concluded from it. Further, they could have pointed out that among the nine selections, three clusters presented themselves. Each cluster of two or three reasons had a larger percentage gap between it and the adjoining cluster than between any of the selections within either cluster.

Average rankings tell us something; gaps between those averages add even more to our understanding.