What Is Statistical Significance and Why Is It Important

Mar 22, 2021 | TIPS & TOOLS, HOW WE MAKE IT

What Is Statistical Significance and Why Is It Important?

When you are conducting A/B testing in marketing, it’s critical for you to understand statistical significance because this information helps you understand whether the results of your experiment are attributed to the changes you made, or due to random chance. A successful conversion driving campaign is said to have statistical significance if it is obvious the campaign led to sales.

Maximizing the number of monetary conversions you receive has to do with knowing what marketing efforts work and which ones do not work. A/B testing is a powerful way to determine this, and every successful marketer understands its importance.

The success of marketing comes down to one thing – Return on Investment (ROI). This is why it is imperative that you know what your ROI is when you’re marketing your business. Conversions directly relate to your ROI, as long as you’re tracking the right type of conversion. A conversion should be what puts money into your pocket – not leads in your hands.

Understanding statistical significance when running A/B testing campaigns will help you understand if your data is reliable enough to make marketing decisions on, and will allow you to be more successful.

Statistical Significance Level

Statistical significance helps you understand risk tolerance and confidence levels. For instance, if you have a statistically significant level of 75%, you are 75% confident the results are due to the changes you made to your campaign. There’s a 25% it was successful due to chance – that’s the risk.

A level that is over 75% means that it was a success. Anything lower, you’re leaning towards not knowing for sure whether what you changed made a difference.

Hypothesizing Results

Statistical significance can prove a hypothesis, which should be part of every campaign.

For instance, if you want to test how changing the color of a BUY NOW button on your site impacts sales, your hypothesis would be “changing the color of the BUY NOW button from blue to green will result in more sales.”

The null hypothesis is the current color – blue. The alternative hypothesis is the color you want to try – green.

To determine if changing the color of the button is statistically significant, you will have two values:

  • P-value – Probability value, which is <0.05.
  • Confidence interval around effect size – upper and lower bounds of what can happen during the campaign.

If your P-value is less than .05, your experiment is statistically significant – your experiment was a success! Good Job.

Here is a visual to help you understand where where the p value is located on a graph (read more on P-value here). When you reach a p value of less than 5% it means that the probability your test will convert in this way is above 95%. So that is pretty reliable data and you can move onto the next experiment.

p value


Dependencies in Statistical Significance

Statistical significance depends on two variables:

Sample Size

  • Effect Size
  • Sample Size

Sample size is who or what you’re testing. You can test products or an audience, depending on what you’re trying to manipulate.

The larger the sample size, the more confidence you will have in your testing. When the sampling is too small, you run the risk of errors.

Effect Size

The effect size measures the results between two sample sets. For instance, if you’re testing the change in color for a BUY NOW button. You may show it to two groups of people of the same number. The number of sales that comes through with the blue and then the green colored button is your effect size. If you have 20 sales from the blue button and 22 sales from the green button, the difference between the 20 and 22 sales is the effect size.

To achieve statistical significance, you need a large sample size AND effect size. You will then have to calculate the difference in results to know if the change is significant.

Calculating Statistical Significance

Mathematicians can calculate statistical significance quite easily, but most marketers are not in the industry due to their math skills. Despite this, they can still use statistical significance in their marketing with a statistical significance calculator.

A statistical significance calculator provides marketers with a way to know if changes in campaigns were successful, so they can maximize conversions. By simply inputting the variables into the calculator, marketers can calculate statistical significance for their sample size and effect size to prove their hypothesis with a statistically significant percentage to show confidence the changes made the difference vs. the results occurring by chance.

Why Statistical Significance Is Important for Businesses

The goal of every marketer should be to increase monetary conversions as much as possible, and understanding their marketing experimentation data shows marketers if changes they make improves their campaigns. Statistical significance is important for businesses because it gives marketers confidence their efforts are headed in the right direction.

Without a reliable data set, people are left to “stab in the dark” rather than know confidently what works and what doesn’t work. Conversions are by chance, but on purpose. Conversions should be on purpose as much as possible to have a successful business.

Digital Dames’ Statistical Significance Calculator

Digital Dames is committed to helping marketers succeed. With easy-to-use tools, such as our statistical significance calculator, we can help take the guesswork out of A/B testing.

A/B testing is a marketing tactic that every single marketer should be doing to maximize conversions. For those who are just getting started with A/B testing, we invite you to read our article “What is A/B Testing? How to Conduct an Effective A/B Test” to learn more about it and how it can help you.

For those who are already familiar with A/B testing but wanted an easier way to explain positive or negative results, statistical significance is the answer. Start using it today to improve your marketing efforts as much as possible to achieve your business goals.


We hope you enjoyed reading this blog post

If you'd like our team at Digital Dames to help you massively improve website traffic and conversions, just book a call.

Post Contributors

Jaclyn Hawtin

Jaclyn Hawtin

Director of Technology

Over a decade of experience in product management, devops, startups, and agile methodologies. Track record of simplifying complex technical processes for cross-functional teams. Proficient in user centered design, UX, IX, UI, IA, user research and data analytics for responsive web, mobile and tablet applications. Incredibly adaptable, fluent with both people and machines.

Mani O'Brien

Mani O'Brien

Director of Marketing

Mani is a senior marketing manager with roots in storytelling. She nerds out on everything data, technology, human behavior and design. Chat with her about UX/UI, marketing funnels, conversion and goal tracking, marketing experimentation and astrology (she’s a Virgo Sun, Aries Rising).

Most Read
Shopify Conversion Tips with Elevar CEO Brad Redding

Shopify Conversion Tips with Elevar CEO Brad Redding

In this episode we have our first podcast guest… we’re excited to welcome Founder & CEO of Elevar, Brad Redding. Brad has specialized in eCommerce and analytics since 2008 when he launched his first SaaS company that was a marketplace matching shoppers and...

4 Steps To A Successful AB Test

4 Steps To A Successful AB Test

Did you know that a CRO expert can easily pinpoint 50 to 150 problems on a website or mobile app? You can't attempt to fix all your problems at once, so how should you prioritize? If you want to convert more customers and increase sales, the answer lies in AB testing,...

Free Analytics 101 Course

Sign up for free access to start your data journey
First Name
This field is for validation purposes and should be left unchanged.
Conversion Path Ep. 12 -- Shopify Conversion Tips With Guest Brad Redding from Elevar Ep. 12 -- Shopify Conversion Tips With Guest Brad Redding from Elevar More