How to Determine Sample Size in A/B Testing

How to Determine Sample Size in A/B Testing

Marketing experimentation (also known as A/B testing or split-testing) should be cornerstone of your digital marketing efforts today. Determining the right sample size is crucial to ensure you are making decisions based on statistically valid results.

Effective A/B testing requires that you present your marketing experiments to enough people (a.k.a. your sample size) to validate your results. So how do you go about calculating sample size in order to get statistically significant results? It depends on four key factors we’ve outlined below.

There Is No Generic Answer to Determine Sample Size

Before we get into exactly how you can determine sample size for A/B testing, it’s important to clarify one thing. There is no generic answer to what you’re looking for. We can’t just give you a number and tell you to stick to it whatever your experiment is. Even if we do give you a fixed number, there’s a pretty high chance that the data you gain from it will hold no value whatsoever.

The Formula

In order to determine sample size, you need to get your hands dirty and work with some math. There is a technical formula that allows you to figure out the exact size of the population necessary to validate your testing. The formula used to calculate this size depends on four factors.

Conversion Rate

You can calculate your conversion rate by dividing the of people that are convinced to purchase by the number of people to whom you presented your product/service. It’s one of the most important factors of digital marketing and the baseline to measure your marketing funnel performance.

Conversion rates can be calculated to give you a sense of your overall visitor-to-customer ratio. For example if every 2 out of 100 website visitors makes a purchase, your conversion rate is 2%. Conversion rates can also be broken down into smaller bit-sized actions within your marketing funnel. For example, if 50 out of 100 website visitors click the Add to Cart button, that is a 50% conversion. In this case you’d want to focus your A/B experiments on these in-between parts of the funnel. In other words, test your Cart page to see if you can increase the number of people who complete a full checkout after Adding to Cart. By increasing the conversion of people complete a  to see if you can increase your overall conversion rate.

Minimum Detectable Effect

The Minimum Detectable Effect, or simply MDE, is a percentage used to denote the difference in percentage. Every organization is expecting some type of improvement from the A/B testing they’re about to run. The MDE is a simple representation of those expectations in the form of a percentage. If you’re looking to improve the conversion rate by 20% from what it is right now, 20% is your MDE.

Statistical Significance

This is a term used in statistics to calculate the number of results from a certain pool of results that did not occur randomly or by chance. It basically helps you understand if you can depend on the results of an A/B Test.

For instance, if the testing has a significant level of 90%, this means that the choice you select after analyzing the results of the test has a 90% probability of being actual results instead of being random. In other words, this also means that there is a 10% chance that the choice you’re making is the wrong one. Statistical significance is another name for reliability, and it can make or break the importance of your A/B testing results (read more about statistical significance in our blog post here).

Statistical Power

Statistical Power is another term that is frequently used in the world of statistics. It basically explains the probability that the experiment or test in question will reach a result. Statistical power does not discuss the factor of right or wrong. It only explains the chances of reaching a result from the two in your sample pool.

Duration of Your A/B Testing

Similar to your sample size, there’s no solid answer to how long your experiment should run for. It depends on a lot of the same factors as the sample size formula we talked about above. It does require some extra values like the number of average daily visitors to your site and the percentage of visitors that are going to be a part of the test. To calculate this number, you can use our duration calculator to conveniently come up with a time span for your A/B testing.

Best Sample Size Calculators

If you’re looking for sample size calculators to take care of the math for you, there are more than a few decent options. Here are a couple of sample size calculators that we like to use.

AB Tasty: AB Tasty is probably the most popular calculation tool for A/B Testing sites on the internet. It not only provides the resources to calculate the right sample size, but also offers a few more neat tools to assist your experimental ventures. You can calculate MDE on a landing page by using the number of visitors and the conversion rate of your site. Apart from that, it also gives an insight into how long your test should run. All you need to offer is an estimate of your daily visitors and the number of variations. Check out AB Tast sample size calculator here.

Optimizely: Optimizely’s A/B Test Size Calculator uses a unique two-tailed sequential likelihood ratio test to calculate and reveal the statistical significance of the test. Unlike AB Tasty, it doesn’t just tell you the results. It also tells you how long the tests should run to finally achieve a higher percentage of statistical significance. Check out Optimizely sample size calculator here.

Closing Thoughts

As small businesses have less traffic yielding less of an impact, they are able to experiment more freely learning through trial and error what works and what doesn’t. However, when it comes to large conglomerates, they don’t really have that luxury. Either they make the right choice, or they lose millions of dollars.

Your marketing isn’t just something you can put at risk. You have to make sure that whatever decision you make is the right one. If you want the help of a professional to boost your marketing, Digital Dames is here to support you. We’re conversion rate optimization agency focused on improving revenue through experimentation.

Most Read
What Is a Good Conversion Rate? See Breakdown by Industry

What Is a Good Conversion Rate? See Breakdown by Industry

“Conversion rate” means everything in marketing. It is the goal of all marketing efforts. The calculation is simple: The percentage of successes from all possibilities. The above definition for conversion rate may sound vague, so here are some examples If your website...

What Is Statistical Significance and Why Is It Important

What Is Statistical Significance and Why Is It Important

What Is Statistical Significance and Why Is It Important? When you are conducting A/B testing in marketing, it's critical for you to understand statistical significance because this information helps you understand whether the results of your experiment are attributed...

How to Setup Conversion Goals in Google Analytics 4

How to Setup Conversion Goals in Google Analytics 4

With the new version of Google Analytics 4 (also known as GA4), the way that you report and view your data is now made easier! In today's video, we will show you how you can set up your conversion goals. Need help with your analytics setup? 👉 Reach out to us for a...

How to Update Google Analytics 4 with Google Tag Manager

How to Update Google Analytics 4 with Google Tag Manager

In today's Lunch & Learn video, we will talk about the things that you need to know about the new Google Analytics 4, released in October 2020. We will cover many topics that will help make it easier for you to get to know and set up Google Analytics 4 with Google...

Jaclyn Hawtin

Jaclyn Hawtin

Senior Data Architect

Over a decade of experience in product management, devops, startups, and agile methodologies. Track record of simplifying complex technical processes for cross-functional teams. Proficient in user centered design, UX, IX, UI, IA, user research and data analytics for responsive web, mobile and tablet applications. Incredibly adaptable, fluent with both people and machines.

More From Dames