Most of us who work on digital products are familiar with the concept of A/B or multivariate testing – the process of exposing users to multiple variations of a design concept and using their aggregate behavior to identify the optimal design, based on a predefined set of metrics. By gathering data across thousands of individual user sessions, multivariate testing can provide a rigorous evidence base for principled decision making. In principle, such data-centric, quantitative research techniques can be highly complementary to the more qualitative, user-centric research techniques typically associated with the UX profession.
But A/B testing isn’t always deployed to the best effect. In my experience, some A/B tests are targeted at genuinely open research questions where the outcome delivers a unique and valuable insight. But there are others where I have thought in hindsight “well, I could have predicted that outcome and saved you the trouble.” Of course, it’s easy to be wise after the event. So here’s a thought: next time you see an A/B test being planned, why don’t you try to predict the outcome? And I mean not just once, but for every test, and then systematically record and measure the accuracy of your own predictions.
IMHO this could generate some valuable insights:
- Qualitative researchers gain a much clearer idea of where their design instincts depart from reality
- Quantitative researchers get to better understand the sorts of problems for which there are existing best practices embedded in collective judgement and UX design expertise
Who knows, we could even train a machine learning model to learn the mapping between problem characteristics and predictability of outcome, and hence create a ‘robot UXer’. I am intrigued by this prospect, and can’t help thinking that others must also have considered this possibility. Does anyone know of any published work exploring this idea?