"The best investment our company has made on analytics" - Benzapp, Bob Martin

How to Determine Sample Size in A/B Testing

Mar 9, 2021 | HOW WE MAKE IT, TIPS & TOOLS

How to Determine Sample Size in A/B Testing

Marketing experimentation (also known as A/B testing or split-testing) should be cornerstone of your digital marketing efforts today. Determining the right sample size is crucial to ensure you are making decisions based on statistically valid results.

Effective A/B testing requires that you present your marketing experiments to enough people (a.k.a. your sample size) to validate your results. So how do you go about calculating sample size in order to get statistically significant results? It depends on four key factors we’ve outlined below.

There Is No Generic Answer to Determine Sample Size

Before we get into exactly how you can determine sample size for A/B testing, it’s important to clarify one thing. There is no generic answer to what you’re looking for. We can’t just give you a number and tell you to stick to it whatever your experiment is. Even if we do give you a fixed number, there’s a pretty high chance that the data you gain from it will hold no value whatsoever.

The Formula

In order to determine sample size, you need to get your hands dirty and work with some math. There is a technical formula that allows you to figure out the exact size of the population necessary to validate your testing. The formula used to calculate this size depends on four factors.

Conversion Rate

You can calculate your conversion rate by dividing the of people that are convinced to purchase by the number of people to whom you presented your product/service. It’s one of the most important factors of digital marketing and the baseline to measure your marketing funnel performance.

Conversion rates can be calculated to give you a sense of your overall visitor-to-customer ratio. For example if every 2 out of 100 website visitors makes a purchase, your conversion rate is 2%. Conversion rates can also be broken down into smaller bit-sized actions within your marketing funnel. For example, if 50 out of 100 website visitors click the Add to Cart button, that is a 50% conversion. In this case you’d want to focus your A/B experiments on these in-between parts of the funnel. In other words, test your Cart page to see if you can increase the number of people who complete a full checkout after Adding to Cart. By increasing the conversion of people complete a  to see if you can increase your overall conversion rate.

Minimum Detectable Effect

The Minimum Detectable Effect, or simply MDE, is a percentage used to denote the difference in percentage. Every organization is expecting some type of improvement from the A/B testing they’re about to run. The MDE is a simple representation of those expectations in the form of a percentage. If you’re looking to improve the conversion rate by 20% from what it is right now, 20% is your MDE.

Statistical Significance

This is a term used in statistics to calculate the number of results from a certain pool of results that did not occur randomly or by chance. It basically helps you understand if you can depend on the results of an A/B Test.

For instance, if the testing has a significant level of 90%, this means that the choice you select after analyzing the results of the test has a 90% probability of being actual results instead of being random. In other words, this also means that there is a 10% chance that the choice you’re making is the wrong one. Statistical significance is another name for reliability, and it can make or break the importance of your A/B testing results (read more about statistical significance in our blog post here).

Statistical Power

Statistical Power is another term that is frequently used in the world of statistics. It basically explains the probability that the experiment or test in question will reach a result. Statistical power does not discuss the factor of right or wrong. It only explains the chances of reaching a result from the two in your sample pool.

Duration of Your A/B Testing

Similar to your sample size, there’s no solid answer to how long your experiment should run for. It depends on a lot of the same factors as the sample size formula we talked about above. It does require some extra values like the number of average daily visitors to your site and the percentage of visitors that are going to be a part of the test. To calculate this number, you can use our duration calculator to conveniently come up with a time span for your A/B testing.

Best Sample Size Calculators

If you’re looking for sample size calculators to take care of the math for you, there are more than a few decent options. Here are a couple of sample size calculators that we like to use.

AB Tasty: AB Tasty is probably the most popular calculation tool for A/B Testing sites on the internet. It not only provides the resources to calculate the right sample size, but also offers a few more neat tools to assist your experimental ventures. You can calculate MDE on a landing page by using the number of visitors and the conversion rate of your site. Apart from that, it also gives an insight into how long your test should run. All you need to offer is an estimate of your daily visitors and the number of variations. Check out AB Tast sample size calculator here.

Optimizely: Optimizely’s A/B Test Size Calculator uses a unique two-tailed sequential likelihood ratio test to calculate and reveal the statistical significance of the test. Unlike AB Tasty, it doesn’t just tell you the results. It also tells you how long the tests should run to finally achieve a higher percentage of statistical significance. Check out Optimizely sample size calculator here.

Closing Thoughts

As small businesses have less traffic yielding less of an impact, they are able to experiment more freely learning through trial and error what works and what doesn’t. However, when it comes to large conglomerates, they don’t really have that luxury. Either they make the right choice, or they lose millions of dollars.

Your marketing isn’t just something you can put at risk. You have to make sure that whatever decision you make is the right one. If you want the help of a professional to boost your marketing, Digital Dames is here to support you. We’re conversion rate optimization agency focused on improving revenue through experimentation.


We hope you enjoyed reading this blog post

If you'd like our team at Digital Dames to help you massively improve website traffic and conversions, just book a call.

Post Contributors

Jaclyn Hawtin

Jaclyn Hawtin

Director of Technology

Over a decade of experience in product management, devops, startups, and agile methodologies. Track record of simplifying complex technical processes for cross-functional teams. Proficient in user centered design, UX, IX, UI, IA, user research and data analytics for responsive web, mobile and tablet applications. Incredibly adaptable, fluent with both people and machines.

Mani O'Brien

Mani O'Brien

Director of Marketing

Mani is a senior marketing manager with roots in storytelling. She nerds out on everything data, technology, human behavior and design. Chat with her about UX/UI, marketing funnels, conversion and goal tracking, marketing experimentation and astrology (she’s a Virgo Sun, Aries Rising).

Most Read
What is Enhanced E-commerce Reporting in Google Analytics?

What is Enhanced E-commerce Reporting in Google Analytics?

E-commerce has become an essential part of modern business, and tracking and analyzing online sales data is crucial for success. Google Analytics is a powerful tool for monitoring website traffic and user behavior, but its basic e-commerce reporting features may not...

What is Enhanced E-commerce Reporting in Google Analytics?

What Are the Best Practices for Optimizing Conversion Rates?

Conversion rates are a critical metric for businesses that want to succeed in the digital age. Whether you're selling products online, generating leads, or promoting your brand, your conversion rate is a measure of how effective your website and marketing campaigns...

What Are the Strategies for Increasing Conversion Rates?

What Are the Strategies for Increasing Conversion Rates?

As a business owner or marketer, one of your primary goals is likely to increase conversion rates. Conversion rates represent the percentage of website visitors who complete a desired action, such as making a purchase, signing up for a newsletter, or filling out a...

How to Improve Your Conversion Rates with A/B Testing

How to Improve Your Conversion Rates with A/B Testing

Improving your conversion rates can make or break your business. After all, you could have a groundbreaking product or service, but if you're unable to convince potential customers to take action, all your hard work will be for nothing. As a savvy business owner,...

What Are the Benefits of Conversion Rate Optimization?

What Are the Benefits of Conversion Rate Optimization?

Looking to increase your profits? If so, you'll want to consider conversion rate optimization (CRO). This powerful strategy involves optimizing your website and marketing materials to improve user experience and encourage more purchases. By implementing CRO tactics,...

Free Analytics 101 Course

Sign up for free access to start your data journey

First Name
This field is for validation purposes and should be left unchanged.
Conversion Path Ep. 12 -- Shopify Conversion Tips With Guest Brad Redding from Elevar Ep. 12 -- Shopify Conversion Tips With Guest Brad Redding from Elevar More