<< by on June 12th, 2013
As Day 2 of Conversion Conference continues, Zee Aganovic, CEO of HiConversion presents 10 inconvenient truths of multivariate and A/B testing:
1. Visitor reactions are unpredictable
Rules based targeting and personalization is very risky, nobody can predict visitor reactions. You must test every option so see how your visitors really act and react.
2. A/B testing is a poor relative of MVT
MVT can be a very arduous process, so most people will avoid this. Every element on the page impacts every other element. So testing elements in silos presents the possibility of making winning elements less effective. MVT allows you to test each element relative to the others.
3. A/B test results aren’t easier to interpret
A/B testing gives you just a yes or no response on the test. MVT testing empowers teams with actionable insights.
4. Testing is about end-to-end results
Page-by-page (iterative) testing is unlikely to yield sustainable results. You cannot assume that one successfully tested element works on every page of a site. Mult-step conversion actions become tricky for testing unless you are able to segment the control group and test group throughout the entire conversion process.
5. Technology Matters
Technology capabilities will impact your strategy and ROI. Better technology will enable best practices and vice versa.
6. Both A/B and MVT are risky
The losers often outweigh the winners. The prospect of losing money is always present. However, intelligent technology can throttle traffic based on predicted performance. Consider using a mature tool that offers an adaptive algorithm approach.
7. Both A/B and MVT are expensive
Consider total cost of ownership and ROI. Factors to consider include cost of tool, implementation, and most importantly (and mostly overlooked), the cost of lost revenue. You are not rich enough to afford cheap solutions. Having good technology pays.
8. Visitor behavior is always changing
There is a median value of visitor behavior, but that can vary as much as 10x. Lead generation is a little more stable, but this is especially true for ecommerce. There are many outside factors that influence a visitors’ decision that are outside of your control. The best way to counteract this is to use reactive technology that will automatically detect these changes and adapt in real time.
9. Lies, damned lies, and statistical significance
Statistical significance and confidence are misleading you. This is due to a lack of real time behavior views. Many times the delay in implementation will render a test void. There is no static solution for a dynamic problem.
10. Best practices are dead, long live best practices
Stop focusing on the “winner”. Look for the variable that consistently performs well and play that all the time as a part of your page.
Key take-aways: you must test. You also must recognize that visitor behavior is always changing and to compensate for that you need intelligent technology.