After running an A/B split test, when the results are in, you always have to make the decision:
Do I switch to the new version or not?
If the difference in conversion is large, the decision is easy: just switch to the best version.
If the difference is small, the decision may be harder, but then again, which way you go doesn’t matter much anyway.
However, I have just done an A/B test where the difference in results is large, but still it’s not clear whether I should stick with version A (my control), or switch to version B (my challenger). Here’s my test: Continue reading