To skip my self-gratifying explanation of why this tool was created and go straight to the tutorial, click here.
Step 1: Admit You Have a Problem.
I am a split testing pansy.
When I speak to internet marketers who tell stories of split testing conquests of conversion rates (CVR) jumping 80% three times in a row, my temperature rises, my hands get clammy, and I do my best to squeeze out an uncomfortable nod-and-smile. Maybe its because I’m ugly. Or maybe its because I’ve forgotten college stats.
Every time I’ve run a split test, I’ve had decent results, but I’ve often pulled the plug due to a creeping feer that my test wasn’t sound. The demons in my head say, “what’s the point of running it if your results aren’t statistically valid anyway?”
Recently, I had the luke-warm honor of proposing a split test to one of our agency clients. If it were for one of my sites, I would have just loaded the variations, closed my eyes and prayed that the wizards behind Optimizely or Visual Website Optimizer had developed a product good enough to tell me when the test was over. Unfortunately, living up to the “consultant” title, I felt obliged to provide realistic expectations about how much traffic we were going to need to achieve statistically significant results for our client.
Step 2: Seek Help.
The first thing I did was hit up the extremely helpful folks at Optimizely to create a spreadsheet model of how their tool actually calculates statistical significance. Someone invested $1.5million for them to create a split testing tool. I assume they know what they’re doing. Pete, one of the creators, sent over an extremely helpful spreadsheet that allowed me to toggle different traffic and conversion numbers to understand how they’d affect statistical significance (note: this sheet is included in the Split Test Planner under the “Results” tab).
In 10 minutes, I learned that determining how much traffic to run through a split test was not so simple.
To predict how much traffic I was going to need to achieve a statistically significant split test, I would need to answer 3 questions -
- What is My Current Conversion Rate? — A page with a 10% CVR is going to need exponentially less data to achieve statistical significance than a page with a 1% CVR.
- How Many Variations Am I Testing? — A test with 4 variations is going to need roughly twice the data of a test with 2 variations. Duh.
- How Much Will I Improve The CVR? — A test that produces a 100% increase in conversion rate will need exponentially less data than a test that only produces a 10% increase.
The first 2 variables are easy to determine. To get your conversion rate, peak into your analytics. To get the number of test variants, peak into your — well, brain. But where the hell do you peek to determine how much of an increase or decrease in CVR your test will yield?
Since you can’t predict how much your test is going to increase the CVR, there’s no way to determine exactly how much data you’re going to need to run a statistically significant split test. Likewise, using a tool like Pete’s would take me ages of entering every possibly conversion rate increase and toggling traffic levels until I hit statistical significance for each one. With no intent of working that hard, I decided to create a tool that quickly let me test statistical significance across a range of CVR improvements at various traffic volumes.
Step 3: Take Control of Your Destiny
FYI: If you watch the video above, you probably don’t need to read the following section.
The Split Test Planner has 2 main sections, Simple Split Test Planner and Advanced Split Test Planner. Below I’ll briefly explain how and when you may want to use them.
Simple Split Test Planner
The Simple Split Test Planner allows you to put in your current CVR and toggle the amount of visitors per variation to see if you’ll achieve statistical significance at 15 different CVR increase levels. Optionally, you may enter the number of variations and estimated cost per click to see the approximate amount of traffic and budget needed for your test.
The Simple Split Test Planner is quick to use and gives you a general feel of what you’re up against based on your sites current conversion rate. Have a great idea that you think will increase your CVR by 5%? Well if your CVR is only 1% as is, you may not want to do that. Hesitating to test your 5% CVR opt-in page? Take a look at how little data you need to get statistically significant results and realize you’re leaving money on the table!
Advanced Split Test Planner
The Advanced Split Test Planner is intended to be a true decision making tool. Unlike the Simple Planner, which shows you whether your test will be statistically significant at 1 traffic volume across 15 CVR improvement levels, the Advanced Tool let’s you see the exact traffic volumes for all 15 improvement levels side by side. The only downside is you’re going to have to work a bit harder and toggle the per-variation traffic volume for each of the 15 improvement levels until they turn green and achieve statistical significance (at a 95% confidence level).
I see two particularly valuable uses for this tool.
First, if you predict a range of CVR improvements for your test (i.e. we expect an improvement of 20–40%), the Advanced tool can help you budget the amount of traffic and dollars you’ll need to allocate for this test.
The second use is for choosing what to test. If you are debating whether to test a 20% predicted change versus a 50% expected change, you can see the exact difference in the costs and traffic necessary to execute the 2 and decide which you’d prefer.
If you are a developer that wants to make a handy web-app around my currently Excel-based tool, please do and post a link in the comments! Likewise, if you have any feedback on how to make the tool better or any issues, please leave them in the comments and we will fix the tool accordingly.
Do you have any split testing war stories? Lessons you’ve learned along the way that make it easier? Statistical principles that make you a more confident marketer?
Share them in the comments below!