On UX and Website Conversion Rate Optimization: Can UX Testing Be Incremental and Innovative?
Design considerations
The most voiced concern for designers is that experiments encourage only small changes and thus stifle creativity. The thought is that experiments are suitable when you want to know which small detail, like button colour, is better, but you should go with your gut when you want to design a new masterpiece.
No doubt these ideas are encouraged by the case studies and examples you’ll find on the internet. They show almost exclusively minimal changes that deliver big wins. There’s little discussion of experimenting with major design projects. You can and should use experimentation for both refinements and drastic redesigns.
The big picture
Running a few experiments will give you a sense of which approaches are the most persuasive to visitors, and what areas of the site are the most influential. Naturally, you’ll want to focus on these areas for maximum effect. Over time this will lead to incrementally smaller changes (Figure 4.1) that might cause you to drift away from the big picture.
Figure 4.1 Marissa Mayer, former vice president of search products and user experience at Google, asked her team to test 41 shades of blue to see which got more clicks.
Incremental improvements
Google’s Scott Huffman recognizes the danger that an experiment-centric business can become too focused on incremental changes and neglect the bigger breakthrough changes: “Testing tools can really motivate the engineering team, but they also can wind up giving them huge incentives to try only small changes. We do want those little improvements, but we also want the jumps outside the box.”
Designer Douglas Bowman felt this frustration in his time at Google: “Yes, it’s true that a team at Google couldn’t decide between two blues, so they’re testing 41 shades between each blue to see which one performs better. I had a recent debate over whether a border should be 3, 4, or 5 pixels wide, and was asked to prove my case.”
Google is right of course to experiment every change—for them, a small drop in conversion would be a loss of millions of dollars. But if you find yourself experimenting something as miniscule as Douglas Bowman’s 5-pixel border, maybe it’s time to reconsider your approach, and aim for bigger wins.
Innovative leaps forward
Incremental design and innovation can go hand in hand. You can run an experiment on anything. You can test an incremental change, or you can test anything from an innovative new feature to an entire redesign. To get the most from your experiments you should be doing both: continually delivering incremental improvements, while never losing sight of the big picture, the big opportunities, and the breakthrough changes.
In Chapter 2, “On Approach,” we discussed the need to ensure some experiment capacity for bigger projects, to allow the room for truly innovative ideas to grow. We’ll talk some more about innovative ideas in Chapter 6, “On Ideas.”
Holistic user experience
Picture Frankenstein’s monster, with parts bolted on here, there, and everywhere. UX designers worry that using experiments as the basis for design decisions creates a “Frankensite,” if you will, whose experience lacks a holistic interconnectedness.
This is a valid concern. As you introduce winners from many experiments, your site’s user experience will no doubt become a little fragmented. If you’re not mindful of cultivating a good user experience in your experiments, your website can eventually become a malformed monster.
Don’t stitch together a monster
You can avoid introducing major inconsistencies by striving incrementally toward a big picture for your website. Of course, this picture needs to be repainted often to adapt to what you’re learning and the direction you’re taking based on new results.
This big picture will likely be something aspirational that reflects the position of your brand and the way your business wants to be seen or positioned in the market. The big picture should be more than a mission statement; it needs concepts and mock-ups so it can easily be conceptualised.
It’s also a good idea, every so often, to run an experiment that simply brings together the parts of your website experience that have become fragmented. (Of course, be sure not to introduce any negatives in doing so.) So, if you see a Frankensite beginning to develop before your eyes, round up all the abnormal pieces and reshape them in one operation to bring them back into an integrated user experience.
Netflix tests across all devices, including desktop, mobile, and apps, and manages to maintain a consistent and simple user interface. Chief Product Officer Neil Hunt shares that Netflix tests almost everything, from calls to action to algorithmic changes in its recommendation engines, to performance components like page load time, streaming time, and quality. Hunt summarizes the company’s learning in three words: “simple trumps complete.” Keeping simplicity at the heart of experiments helps Netflix deliver a smooth and adaptable user experience across devices (Figure 4.2), so whether you view your movies or rental list on a desktop, an iPhone, or a PlayStation 3, it always feels like Netflix and one coherent journey.
Figure 4.2. Netflix experiments almost everything across all devices, and still maintains a simple and consistent user experience. (http://www.quora.com/What-types-of-things-does-Netflix-A-B-test-aside-from-member-sign-up)
Avoid consistency hobgoblins
While delivering a holistic user experience is important, don’t let fear of inconsistencies hold back your experiments. You can always return to an inconsistency, but you can’t return to something you haven’t yet discovered because an idealistic view on user experience prevented you from running an experiment in the first place.
Give rise to generalizations
As you experiment with incremental improvements, small changes may give rise to generalizations. An experiment to determine whether a border should be 3, 4, or 5 pixels may be a tiny experiment in itself, but could lead to a fundamental insight that can be applied multiple times to many borders. You could create principles to capture these generalizations and use them to inform future design decisions.