What's your executive dashboard hiding?
In our sound byte culture we love condensed statistics, magic values which will let us know how many customers will buy again or whether employees are engaged in their work—all at an easily compared glance. While summary information such as means (averages) is very useful, it can also obscure some critical details and trends within your data. When many factors are condensed into a single index value for executive dashboards, even more can be hidden.
Let's consider a set of customer service center rating questions, using a scale from 1 to 5. In real surveys these will have different means, but for illustration ours are all going to have a mean right around 3.0:
If we only use this information, we might conclude that the different factors are all rated similarly by our customers. However, if we also look at the standard deviation we get a slightly different picture:
The above chart has the mean as the centerline of each bar, and then the length of the bar indicates the standard deviation. A standard deviation tells you the range within which 68% of your respondents fall. In our example we can see that there was quite a bit of variation on all the ratings except Technical knowledge, where we have a much more tightly clustered set of responses.
So, that was a little more information—better than the mean alone—but what else might be hidden in those similar means and standard deviations? Quite a bit, as you can see when you run percentage breakdowns:
As you can see, there are significantly different profiles for each of the factors, including one where the mean represents only 1/3 the number of people as the others. In your own data, the mean and other dashboard statistics may be a useful starting point, but they cannot replace regular reviews where you drill down deeper. For this type of performance rating, a core set of statistics to review each month or quarter includes:
- Standard deviation
- Total number of responses
- Frequency (counts or percents) for each response
- Trends over time
These will give you a fairly complete picture of what people think (without requiring a statistics book). Remember though, they still don't tell you anything about why people feel the way they do!
I always hear your voice in my head ‘if it is taking you too long, there is a simpler way to do it!’
Market Research Analyst
Alberta Motor Association