If you’ve heard the term “statistical significance” a lot lately you’re not alone. The phrase gets thrown around a good deal these days especially during and after the most recent presidential election. You might have heard “margin of error” and “statistical dead heat” several times as well. Statistical significance isn’t just another buzzword, but it is greatly misunderstood. Here’s what you need to know about the term and how misunderstandings of its meaning could affect your business.
Sure, It’s Significant But Is It Important?
Margin of error and statistical dead heat actually refer to the same concept of statistical significance. But what does statistical significance truly mean? Anything deemed statistically significant is saying that the probability of two populations (e.g. two processes, two production lots, or the size of two groups of voters that intend to vote for one candidate or another) would be the same if we had the data on the complete population. It’s important to remember that all projections about the truth regarding a population from sample data is based on probability.
While many people get excited when they learn a finding is statistically significant, before they get too excited, they should consider if the amount of difference they found is really enough to change what actions they take — is the difference practically significant. It’s important to take note that while analysts may use “significant” to describe an important finding, the meaning of the word in the world of statistics is simply that it appears likely that there really is some level of difference between the two or more things that they are comparing — not necessarily that there is a lot of difference.
Statistically significant findings tell us that we can be confident that the result is real, rather than a mere fluke. So, before attempting to draw insights from a statistically significant finding, be sure to take note of what the results of the finding are telling you. Are the findings revealing information that should factor into future business decisions? Or are they merely suggesting a relationship between two populations at this time? Drawing conclusions before they’re ready to be drawn can lead to costly mistakes and risky business decisions.
Ready to use statistical significance to run better experiments for continuous improvement? Then sign up for our upcoming workshop Design & Analysis of Experiment, learn more here.