What Is Statistical Significance and Why Is It Important

What Is Statistical Significance and Why Is It Important?

When you are conducting A/B testing in marketing, it’s critical for you to understand statistical significance because this information helps you understand whether the results of your experiment are attributed to the changes you made, or due to random chance. A successful conversion driving campaign is said to have statistical significance if it is obvious the campaign led to sales.

Maximizing the number of monetary conversions you receive has to do with knowing what marketing efforts work and which ones do not work. A/B testing is a powerful way to determine this, and every successful marketer understands its importance.

The success of marketing comes down to one thing – Return on Investment (ROI). This is why it is imperative that you know what your ROI is when you’re marketing your business. Conversions directly relate to your ROI, as long as you’re tracking the right type of conversion. A conversion should be what puts money into your pocket – not leads in your hands.

Understanding statistical significance when running A/B testing campaigns will help you understand if your data is reliable enough to make marketing decisions on, and will allow you to be more successful.

Statistical Significance Level

Statistical significance helps you understand risk tolerance and confidence levels. For instance, if you have a statistically significant level of 75%, you are 75% confident the results are due to the changes you made to your campaign. There’s a 25% it was successful due to chance – that’s the risk.

A level that is over 75% means that it was a success. Anything lower, you’re leaning towards not knowing for sure whether what you changed made a difference.

Hypothesizing Results

Statistical significance can prove a hypothesis, which should be part of every campaign.

For instance, if you want to test how changing the color of a BUY NOW button on your site impacts sales, your hypothesis would be “changing the color of the BUY NOW button from blue to green will result in more sales.”

The null hypothesis is the current color – blue. The alternative hypothesis is the color you want to try – green.

To determine if changing the color of the button is statistically significant, you will have two values:

  • P-value – Probability value, which is <0.05.
  • Confidence interval around effect size – upper and lower bounds of what can happen during the campaign.

If your P-value is less than .05, your experiment is statistically significant – your experiment was a success! Good Job.

Here is a visual to help you understand where where the p value is located on a graph (read more on P-value here). When you reach a p value of less than 5% it means that the probability your test will convert in this way is above 95%. So that is pretty reliable data and you can move onto the next experiment.


Dependencies in Statistical Significance

Statistical significance depends on two variables:

Sample Size

  • Effect Size
  • Sample Size

Sample size is who or what you’re testing. You can test products or an audience, depending on what you’re trying to manipulate.

The larger the sample size, the more confidence you will have in your testing. When the sampling is too small, you run the risk of errors.

Effect Size

The effect size measures the results between two sample sets. For instance, if you’re testing the change in color for a BUY NOW button. You may show it to two groups of people of the same number. The number of sales that comes through with the blue and then the green colored button is your effect size. If you have 20 sales from the blue button and 22 sales from the green button, the difference between the 20 and 22 sales is the effect size.

To achieve statistical significance, you need a large sample size AND effect size. You will then have to calculate the difference in results to know if the change is significant.

Calculating Statistical Significance

Mathematicians can calculate statistical significance quite easily, but most marketers are not in the industry due to their math skills. Despite this, they can still use statistical significance in their marketing with a statistical significance calculator.

A statistical significance calculator provides marketers with a way to know if changes in campaigns were successful, so they can maximize conversions. By simply inputting the variables into the calculator, marketers can calculate statistical significance for their sample size and effect size to prove their hypothesis with a statistically significant percentage to show confidence the changes made the difference vs. the results occurring by chance.

Why Statistical Significance Is Important for Businesses

The goal of every marketer should be to increase monetary conversions as much as possible, and understanding their marketing experimentation data shows marketers if changes they make improves their campaigns. Statistical significance is important for businesses because it gives marketers confidence their efforts are headed in the right direction.

Without a reliable data set, people are left to “stab in the dark” rather than know confidently what works and what doesn’t work. Conversions are by chance, but on purpose. Conversions should be on purpose as much as possible to have a successful business.

Digital Dames’ Statistical Significance Calculator

Digital Dames is committed to helping marketers succeed. With easy-to-use tools, such as our statistical significance calculator, we can help take the guesswork out of A/B testing.

A/B testing is a marketing tactic that every single marketer should be doing to maximize conversions. For those who are just getting started with A/B testing, we invite you to read our article “What is A/B Testing? How to Conduct an Effective A/B Test” to learn more about it and how it can help you.

For those who are already familiar with A/B testing but wanted an easier way to explain positive or negative results, statistical significance is the answer. Start using it today to improve your marketing efforts as much as possible to achieve your business goals.

Most Read
What Is a Good Conversion Rate? See Breakdown by Industry

What Is a Good Conversion Rate? See Breakdown by Industry

“Conversion rate” means everything in marketing. It is the goal of all marketing efforts. The calculation is simple: The percentage of successes from all possibilities. The above definition for conversion rate may sound vague, so here are some examples If your website...

How to Determine Sample Size in A/B Testing

How to Determine Sample Size in A/B Testing

How to Determine Sample Size in A/B Testing Marketing experimentation (also known as A/B testing or split-testing) should be cornerstone of your digital marketing efforts today. Determining the right sample size is crucial to ensure you are making decisions based on...

How to Setup Conversion Goals in Google Analytics 4

How to Setup Conversion Goals in Google Analytics 4

With the new version of Google Analytics 4 (also known as GA4), the way that you report and view your data is now made easier! In today's video, we will show you how you can set up your conversion goals. Need help with your analytics setup? 👉 Reach out to us for a...

How to Update Google Analytics 4 with Google Tag Manager

How to Update Google Analytics 4 with Google Tag Manager

In today's Lunch & Learn video, we will talk about the things that you need to know about the new Google Analytics 4, released in October 2020. We will cover many topics that will help make it easier for you to get to know and set up Google Analytics 4 with Google...

Jaclyn Hawtin

Jaclyn Hawtin

Senior Data Architect

Over a decade of experience in product management, devops, startups, and agile methodologies. Track record of simplifying complex technical processes for cross-functional teams. Proficient in user centered design, UX, IX, UI, IA, user research and data analytics for responsive web, mobile and tablet applications. Incredibly adaptable, fluent with both people and machines.

More From Dames