Thoughtful testing
But in order for me to validate all of this information, that’s where AB testing comes in. I can’t really be making a change and then looking, oh, well, you know, what type of an impact did it have? Because just looking at my analytics won’t tell me much. AB testing is gonna help me validate exactly whether or not that change had an impact on my visitors.
And remember, again, even though we’ve done the research, and we’ve found a problem area, it doesn’t mean that my solution is necessarily going to resonate with the visitors. That’s why I really need to AB test it in order to validate it even further.
Ask questions
The other thing is that a lot of times, people say AB test everything. AB test this, AB test that. But that means that I’m really not conducting any research. I’m just randomly choosing anything. I get to bring in any group of designers, I can bring in any group of people, and they’ll have a hundred different suggestions that I should do on a website. Does that mean that I should AB test all of those things? ‘Cause AB testing takes time, it takes effort, it takes development. I wanna make sure that whatever I’m testing, it’s grounded in some sort of research. I’m looking for a repeatable, sustainable process. And the only way I can get to that process is by doing a lot of these different things. I’m trying to really conduct all this research.
You’ll notice that a lot of AB testing tools report that only 15 to 20% of the tests that they launch on their platforms yield positive results. That’s crazy. 15 to 20%. This is Optimizely, VWO, Google Optimize before it’s, you know, well, obviously, it’s currently sunsetting or already sunset, but the idea is that a lot of tests don’t actually perform well. And the reason why is they’re not grounded in research. They’re not being vetted. They’re just random experiments that people just think of. Let’s just test this and see how it performs.
Now I will say that if you’re going to add an element on your website, you do wanna make sure that that actually gets tested. The reason is because, again, you don’t wanna add something onto your website that might impact performance, that might impact your visitors without necessarily kind of figuring out whether or not it’s going to have a positive result, yield a positive result for you.
Brush up on your stats
The third point that I wanna make is that you wanna brush up on your stats. So, you don’t need a degree in statistics. I definitely don’t have a degree in statistics I’ll tell you that. But you need to understand some of the basics.
Did your test run long enough?
First of all, you wanna make sure that your test runs long enough. I talked a little bit about those sensational case studies. A lot of times, when you dig into the data of those sensational case studies, they haven’t run long enough, they haven’t reached a significant result. So you wanna make sure you avoid false positives and negatives in order to make sure that you’re reaching a specific sample size. Now, sometimes your sample size might just be three days.
Run your test for a minimum of a week
We always recommend that you run an experiment for at least a minimum of a week in order for you to be able to make sure that the test hits an entire business cycle, right? Because again, when I’m like thinking about my customers and when they’re coming, I wanna think about weekends, I wanna think about the entire week. So we always say a minimum of a week. We had one client, for example, that we’d always see a surge of conversions on the 15th of the month and the end of the month, and those are payday. So again, you wanna make sure that you ran it long enough, a minimum of a week.
Look at an 85% — 90% chance to win
And when you’re thinking about the chance to win, I always say, look at an 85% to 90% chance to win. And the reason why, again, ’cause there’s always still a chance that it actually might be a false positive, you wanna make sure you make that chance even lower. So just make sure that you look at all those stats, you consider all of those areas, you conduct a sample size, there’s lots of calculators out there. You can just look at your sample size, and figure out what the sample size is before you actually run the experiment in order for you to make sure that you don’t run into any issues when it comes to the stats of the experiment.
