The Truth

HOTH Split Test Planner — Free Tool That Makes Split Testing Less Scary.

To skip my self-gratifying expla­na­tion of why this tool was cre­ated and go straight to the tuto­r­ial, click here.

Step 1: Admit You Have a Problem.

I am a split test­ing pansy.

When I speak to inter­net mar­keters who tell sto­ries of split test­ing con­quests of con­ver­sion rates (CVR) jump­ing 80% three times in a row, my tem­per­a­ture rises, my hands get clammy, and I do my best to squeeze out an uncom­fort­able nod-and-smile. Maybe its because I’m ugly. Or maybe its because I’ve for­got­ten col­lege stats.

Every time I’ve run a split test, I’ve had decent results, but I’ve often pulled the plug due to a creep­ing feer that my test wasn’t sound. The demons in my head say, “what’s the point of run­ning it if your results aren’t sta­tis­ti­cally valid anyway?”

Recently, I had the luke-warm honor of propos­ing a split test to one of our agency clients. If it were for one of my sites, I would have just loaded the vari­a­tions, closed my eyes and prayed that the wiz­ards behind Opti­mizely or Visual Web­site Opti­mizer had devel­oped a prod­uct good enough to tell me when the test was over. Unfor­tu­nately, liv­ing up to the “con­sul­tant” title, I felt obliged to pro­vide real­is­tic expec­ta­tions about how much traf­fic we were going to need to achieve sta­tis­ti­cally sig­nif­i­cant results for our client.

Step 2: Seek Help.

The first thing I did was hit up the extremely help­ful folks at Opti­mizely to cre­ate a spread­sheet model of how their tool actu­ally cal­cu­lates sta­tis­ti­cal sig­nif­i­cance. Some­one invested $1.5million for them to cre­ate a split test­ing tool. I assume they know what they’re doing. Pete, one of the cre­ators, sent over an extremely help­ful spread­sheet that allowed me to tog­gle dif­fer­ent traf­fic and con­ver­sion num­bers to under­stand how they’d affect sta­tis­ti­cal sig­nif­i­cance (note: this sheet is included in the Split Test Plan­ner under the “Results” tab).

In 10 min­utes, I learned that deter­min­ing how much traf­fic to run through a split test was not so simple.

To pre­dict how much traf­fic I was going to need to achieve a sta­tis­ti­cally sig­nif­i­cant split test, I would need to answer 3 questions -

1. What is My Cur­rent Con­ver­sion Rate? — A page with a 10% CVR is going to need expo­nen­tially less data to achieve sta­tis­ti­cal sig­nif­i­cance than a page with a 1% CVR.
2. How Many Vari­a­tions Am I Test­ing? — A test with 4 vari­a­tions is going to need roughly twice the data of a test with 2 vari­a­tions. Duh.
3. How Much Will I Improve The CVR? — A test that pro­duces a 100% increase in con­ver­sion rate will need expo­nen­tially less data than a test that only pro­duces a 10% increase.

The first 2 vari­ables are easy to deter­mine. To get your con­ver­sion rate, peak into your ana­lyt­ics. To get the num­ber of test vari­ants, peak into your — well, brain. But where the hell do you peek to deter­mine how much of an increase or decrease in CVR your test will yield?

You don’t.

Since you can’t pre­dict how much your test is going to increase the CVR, there’s no way to deter­mine exactly how much data you’re going to need to run a sta­tis­ti­cally sig­nif­i­cant split test. Like­wise, using a tool like Pete’s would take me ages of enter­ing every pos­si­bly con­ver­sion rate increase and tog­gling traf­fic lev­els until I hit sta­tis­ti­cal sig­nif­i­cance for each one. With no intent of work­ing that hard, I decided to cre­ate a tool that quickly let me test sta­tis­ti­cal sig­nif­i­cance across a range of CVR improve­ments at var­i­ous traf­fic volumes.

Step 3: Take Con­trol of Your Destiny

FYI: If you watch the video above, you prob­a­bly don’t need to read the fol­low­ing section.

The Split Test Plan­ner has 2 main sec­tions, Sim­ple Split Test Plan­ner and Advanced Split Test Plan­ner. Below I’ll briefly explain how and when you may want to use them.

Sim­ple Split Test Planner

The Sim­ple Split Test Plan­ner allows you to put in your cur­rent CVR and tog­gle the amount of vis­i­tors per vari­a­tion to see if you’ll achieve sta­tis­ti­cal sig­nif­i­cance at 15 dif­fer­ent CVR increase lev­els. Option­ally, you may enter the num­ber of vari­a­tions and esti­mated cost per click to see the approx­i­mate amount of traf­fic and bud­get needed for your test.

The Sim­ple Split Test Plan­ner is quick to use and gives you a gen­eral feel of what you’re up against based on your sites cur­rent con­ver­sion rate. Have a great idea that you think will increase your CVR by 5%? Well if your CVR is only 1% as is, you may not want to do that. Hes­i­tat­ing to test your 5% CVR opt-in page? Take a look at how lit­tle data you need to get sta­tis­ti­cally sig­nif­i­cant results and real­ize you’re leav­ing money on the table!

Advanced Split Test Planner

The Advanced Split Test Plan­ner is intended to be a true deci­sion mak­ing tool. Unlike the Sim­ple Plan­ner, which shows you whether your test will be sta­tis­ti­cally sig­nif­i­cant at 1 traf­fic vol­ume across 15 CVR improve­ment lev­els, the Advanced Tool let’s you see the exact traf­fic vol­umes for all 15 improve­ment lev­els side by side. The only down­side is you’re going to have to work a bit harder and tog­gle the per-variation traf­fic vol­ume for each of the 15 improve­ment lev­els until they turn green and achieve sta­tis­ti­cal sig­nif­i­cance (at a 95% con­fi­dence level).

I see two par­tic­u­larly valu­able uses for this tool.

First, if you pre­dict a range of CVR improve­ments for your test (i.e. we expect an improve­ment of 20–40%), the Advanced tool can help you bud­get the amount of traf­fic and dol­lars you’ll need to allo­cate for this test.

The sec­ond use is for choos­ing what to test. If you are debat­ing whether to test a 20% pre­dicted change ver­sus a 50% expected change, you can see the exact dif­fer­ence in the costs and traf­fic nec­es­sary to exe­cute the 2 and decide which you’d prefer.

Con­tribute!

If you are a devel­oper that wants to make a handy web-app around my cur­rently Excel-based tool, please do and post a link in the com­ments! Like­wise, if you have any feed­back on how to make the tool bet­ter or any issues, please leave them in the com­ments and we will fix the tool accordingly.

Do you have any split test­ing war sto­ries? Lessons you’ve learned along the way that make it eas­ier? Sta­tis­ti­cal prin­ci­ples that make you a more con­fi­dent marketer?

Share them in the com­ments below!