How to Determine Sample Size in A/B Testing

Mar 9, 2021 | HOW WE MAKE IT, TIPS & TOOLS

How to Determine Sample Size in A/B Testing

Marketing experimentation (also known as A/B testing or split-testing) should be cornerstone of your digital marketing efforts today. Determining the right sample size is crucial to ensure you are making decisions based on statistically valid results.

Effective A/B testing requires that you present your marketing experiments to enough people (a.k.a. your sample size) to validate your results. So how do you go about calculating sample size in order to get statistically significant results? It depends on four key factors we’ve outlined below.

There Is No Generic Answer to Determine Sample Size

Before we get into exactly how you can determine sample size for A/B testing, it’s important to clarify one thing. There is no generic answer to what you’re looking for. We can’t just give you a number and tell you to stick to it whatever your experiment is. Even if we do give you a fixed number, there’s a pretty high chance that the data you gain from it will hold no value whatsoever.

The Formula

In order to determine sample size, you need to get your hands dirty and work with some math. There is a technical formula that allows you to figure out the exact size of the population necessary to validate your testing. The formula used to calculate this size depends on four factors.

Conversion Rate

You can calculate your conversion rate by dividing the of people that are convinced to purchase by the number of people to whom you presented your product/service. It’s one of the most important factors of digital marketing and the baseline to measure your marketing funnel performance.

Conversion rates can be calculated to give you a sense of your overall visitor-to-customer ratio. For example if every 2 out of 100 website visitors makes a purchase, your conversion rate is 2%. Conversion rates can also be broken down into smaller bit-sized actions within your marketing funnel. For example, if 50 out of 100 website visitors click the Add to Cart button, that is a 50% conversion. In this case you’d want to focus your A/B experiments on these in-between parts of the funnel. In other words, test your Cart page to see if you can increase the number of people who complete a full checkout after Adding to Cart. By increasing the conversion of people complete a  to see if you can increase your overall conversion rate.

Minimum Detectable Effect

The Minimum Detectable Effect, or simply MDE, is a percentage used to denote the difference in percentage. Every organization is expecting some type of improvement from the A/B testing they’re about to run. The MDE is a simple representation of those expectations in the form of a percentage. If you’re looking to improve the conversion rate by 20% from what it is right now, 20% is your MDE.

Statistical Significance

This is a term used in statistics to calculate the number of results from a certain pool of results that did not occur randomly or by chance. It basically helps you understand if you can depend on the results of an A/B Test.

For instance, if the testing has a significant level of 90%, this means that the choice you select after analyzing the results of the test has a 90% probability of being actual results instead of being random. In other words, this also means that there is a 10% chance that the choice you’re making is the wrong one. Statistical significance is another name for reliability, and it can make or break the importance of your A/B testing results (read more about statistical significance in our blog post here).

Statistical Power

Statistical Power is another term that is frequently used in the world of statistics. It basically explains the probability that the experiment or test in question will reach a result. Statistical power does not discuss the factor of right or wrong. It only explains the chances of reaching a result from the two in your sample pool.

Duration of Your A/B Testing

Similar to your sample size, there’s no solid answer to how long your experiment should run for. It depends on a lot of the same factors as the sample size formula we talked about above. It does require some extra values like the number of average daily visitors to your site and the percentage of visitors that are going to be a part of the test. To calculate this number, you can use our duration calculator to conveniently come up with a time span for your A/B testing.

Best Sample Size Calculators

If you’re looking for sample size calculators to take care of the math for you, there are more than a few decent options. Here are a couple of sample size calculators that we like to use.

AB Tasty: AB Tasty is probably the most popular calculation tool for A/B Testing sites on the internet. It not only provides the resources to calculate the right sample size, but also offers a few more neat tools to assist your experimental ventures. You can calculate MDE on a landing page by using the number of visitors and the conversion rate of your site. Apart from that, it also gives an insight into how long your test should run. All you need to offer is an estimate of your daily visitors and the number of variations. Check out AB Tast sample size calculator here.

Optimizely: Optimizely’s A/B Test Size Calculator uses a unique two-tailed sequential likelihood ratio test to calculate and reveal the statistical significance of the test. Unlike AB Tasty, it doesn’t just tell you the results. It also tells you how long the tests should run to finally achieve a higher percentage of statistical significance. Check out Optimizely sample size calculator here.

Closing Thoughts

As small businesses have less traffic yielding less of an impact, they are able to experiment more freely learning through trial and error what works and what doesn’t. However, when it comes to large conglomerates, they don’t really have that luxury. Either they make the right choice, or they lose millions of dollars.

Your marketing isn’t just something you can put at risk. You have to make sure that whatever decision you make is the right one. If you want the help of a professional to boost your marketing, Digital Dames is here to support you. We’re conversion rate optimization agency focused on improving revenue through experimentation.

We hope you enjoyed reading this blog post

If you’d like our team at Digital Dames to help you massively improve website traffic and conversions, just book a call.

Post Contributors

Jaclyn Hawtin

Jaclyn Hawtin

Senior Data Architect

Over a decade of experience in product management, devops, startups, and agile methodologies. Track record of simplifying complex technical processes for cross-functional teams. Proficient in user centered design, UX, IX, UI, IA, user research and data analytics for responsive web, mobile and tablet applications. Incredibly adaptable, fluent with both people and machines.

Mani O'Brien

Mani O'Brien

Conversion optimization manager

Mani is a senior marketing manager with roots in storytelling. She nerds out on everything data, technology, human behavior and design. Chat with her about UX/UI, marketing funnels, conversion and goal tracking, marketing experimentation and astrology (she’s a Virgo Sun, Aries Rising).

Most Read
Black Lives Matter. The Ways We Commit to Antiracism as a Company

Black Lives Matter. The Ways We Commit to Antiracism as a Company

At Digital Dames we believe that Black Lives Matter. We have always held women, minorities and otherwise disenfranchised communities near to our hearts as we grow this company. With that, we recognize that there is much unlearning to do in order to unpack our own...

Mobile App Marketing: iOS 14.5 Apple’s App Tracking Transparency

Mobile App Marketing: iOS 14.5 Apple’s App Tracking Transparency

Mobile App Marketing: The Impact of iOS 14.5 Apple App Tracking Transparency Wow! Apple has rolled out its new privacy/consent updates with iOS 14.5 giving consumers more control over what data they share with mobile apps. What does this mean for mobile app marketing?...

What Is a Good eCommerce Conversion Rate?

What Is a Good eCommerce Conversion Rate?

What Is a Good eCommerce Conversion Rate? Are you looking at your conversion rates for landing pages and your eCommerce site overall wondering if they are good enough? You’re not the only one! The majority of eCommerce business owners have a conversion rate they would...

How to Increase Conversion Rate on Shopify

How to Increase Conversion Rate on Shopify

How to Increase Conversion Rate on Shopify The Shopify platform is wildly successful, but how do you go about improving conversions? Customers love Shopify. In fact, not only has Shopify managed to become one of the world's top e-commerce platforms, attracting...