Originally published August 2021
Statistical significance: just these words alone are often enough to make marketers anxious when diving into the findings of quantitative research reports. In our experience the concept of statistical significance is a common cause of confusion, and for good reason. If you’re morbidly curious, just google it and check back in after you’re adequately overwhelmed and/or confused by all the technical vocabulary and mathematical expressions.
If you somehow manage to navigate through the maze of scientific terminology, there are still so many questions that remain. Are statistically significant findings the only meaningful ones? If it’s significant at a 90% confidence but not at 95%, should I ditch that analysis? If my sample size is less than 30, are the findings meaningless? These questions are all good ones, and deserve answers on a case-by-case basis. However, in this spotlight we’d like to lay down some ground rules for interpreting significant and non-significant findings without asking you to go and get a PhD in statistics.
There is a difference between what is statistically significant, and what is meaningful.
In the historic days before the internet, getting enough data to make business decisions confidently was more difficult. These days we’ve solved the "not enough data" problem, but now we have an emergent problem: More data isn’t necessarily better. Data is impartial to the truth. A lot of data means exactly nothing, yet one small number could be worth a billion dollars. So—how do we find the meaningful nuggets of signal when data can contain a lot of noise? Oftentimes we overweight the importance of statistically significant data that is meaningless, and underweight the importance of a non-significant finding that may directly answer your business question.
The statistical test is the detective, not the judge.
We can’t tell you how many analysts we’ve met (sometimes we catch ourselves!) that scan through reams of data tables and highlight all the statistically significant differences as the first step in doing analysis. At Advantage Research, we’ve made this a sort of no-no in our culture. Why? Because indiscriminately filling your brain with tons of (mostly meaningless) information and then trying to make sense of it is a heck of a lot harder than being scientific—that is, formulating a hypothesis, and then testing it by examining the data that can confirm or reject the hypothesis. Let the computers do what they’re good at (remembering lots of disparate stuff), and let’s do what we’re good at (finding the patterns and meaning).
Be skeptical, even of what you feel is certain.
We live in a world bursting at the seams with data, and equipping yourself with the wrong knowledge is the first step towards going out of business. This isn’t to undermine the importance of being decisive in our business decision making; making educated decisions based on research is far better than shooting from the hip using hunches and anecdotes! However, holding on to what you think you know makes you resistant to learning, and the world is always changing. Remain very open to the possibility that you could be wrong, and always ask yourself how you could be, especially when you feel certain.
Assess the importance of the business decision in question.
How confident do you need to be? Big decisions require big evidence, smaller decisions do not! In statistical hypothesis testing, confidence is expensive and takes a long time to build. Sometimes, directional findings from small sample sizes wind up being absolutely correct. On the contrary, it’s good to be confident that a bus isn’t coming before you cross the street, even if you’re wrong and one isn’t!
We hope this clears up some of the confusion around statistical significance, and demystifies it a bit. It’s merely a tool that we can use in research to build confidence, and is far from the only consideration we should make in doing our analysis!
If you need help with quantitative research for your organization, Advantage Research can recommend an approach that is right for you. Contact us today to start the conversation!