Six questions to ask any statistic

By Ann Ray on February 24, 2006 | Last updated April 25, 2008
Permalink | Comments (0)

Numbers have an apparent precision, one often given far more credibility than they deserve. Here are six questions to ask any statistic, whether it's one you're generating or using.

1. Who does this represent?
If the statistic is about the opinions of Fortune executives, it's probably not going to be useful for predicting the actions of rural farmers. While this is obvious, the problem with many statistics is that they don't tell you anything about who answered the survey.

2. How many people does it represent?
Some studies, particularly ones which are very expensive to conduct, will draw conclusions from less than 100 participants. Others will have thousands of respondents. The main factor for gauging reliability is the portion of the target population (who the statistic represents) that answered. If you do come across a statistic based on very few people, it's not necessarily wrong, it's just more hypothesis than fact until more research is done to confirm it.

3. How were people reached?
So the statistic claims to represent our Fortune executives, but when you dig a little deeper you discover the data was gathered via phone and mall intercept interviews. Would you still think it represented executives? Selecting the right method for collecting answers is as much art as science, but it's an important aspect to check.

4. How were questions phrased?
We've all seen "polls" which were designed to produce a certain result. Make sure the research was done in a properly neutral manner.

5. Who commissioned the research?
Even if they have the best intentions for pure research, a special interest group has a "truth" they believe, and sometimes a bias is introduced into the questions and analysis. At the least, try to find a study from the other side for comparison.

6. When was the study conducted?
Some really old statistics are running around. Some have been cited repeatedly over the decades, so the article date listed as the source may not be the original study date.

In practice, of course, you're not always able to ask all these questions. However, the more holes there are in a statistic's back-story, the more risk you run if you rely on it alone as a "fact." Even your own custom survey research should always be balanced by information such as industry studies, competitive positions, experience, and financial data.