A Primer on Multivariate Testing

In my last post on testing I received some great feedback from my readers. One such feedback was around “describing” the principles of MVT. The question was “It would be great if you would be able to get into further detail on the principles behind this (MVT) form of testing”. This question has become the motivation behind this persuasion. In this post  we will recap  the Why’s of testing and experimentation and then jump into the principles of Multivariate Testing and how it impacts business performance. We will also look at designing experiments with Multivariate Testing (Levels, Factors, and Orthogonal Arrays).

Sounds complicated? Don’t worry. The goal of this persuasion is to provide you with a straight forward description of Multivariate Testing so you can walk the walk and talk the talk in real world scenarios. If you stick with me,  I am sure  you will elevate your learning around MVT.

It’s worth pointing out that proactive testing and experimentation offers a sustainable competitive advantage. Majority of the digital properties evolve in response to their competition and that evolution is slow (and poor). Testing and Experimentation serves as an enabler to “bring the organization along” to:

Complement the “art” of web design (right brain) with “science” (left brain) by validating design decisions with statistically significant findings by identifying factors influencing your business objectives. More importantly controlled experiments can help you quantify the variables which have a significant effect on your desired response by screening out analysis of insignificant variables. (Correlation is not Causation)

Optimize site improvements at any stage  of the organizational evolution. If you are new to testing, you can hit the ground running by adopting simple frameworks; if you are already performing some testing you can turbo charge your improvement process by leveraging principles of successful experimental design techniques illustrated in this persuasion.


Less is More


Magnify the impact of Web Analytics. This is my personal favorite as it enables your group to be strategic (vs. ad hoc) and empowers you to accelerate the evolution of your digital presence through proactive discovery.



How do you define Multivariate Testing and What are it’s benefits?

This form of testing involves studying the effects of multiple factors simultaneously  in an efficient way so as to isolate the most important elements and combinations on a desired outcome. I discussed this at a high level in my prior post. This form of testing is ideally suited to Web Analytics (more than any other industry).

It also yields a significantly better set of results with fraction of the effort required by other methods.

Finally, the results for this form of testing are almost immediate with neglible costs when compared to other testing approaches.

…. and its advantages?

Efficiency: By significantly fewer testing iterations than A/B one can take less time and perform test and learn of key variables that drive conversion with minimal cost.

Optimizing Combination of Variables: Allowing for testing of possible interaction effects between variables.

As an example, here is how you can consider performing a Multivariate Test on Google Analytics homepage. In this case user can potentially test Position of Logo, Navigation Bar Type, Splash Page, and Selection of Clients to drive their desired macro (or micro) outcomes.

For example, Google may want to know which of the factors/levels (see description below) contributes to signing-up for their Analytics platform. (Is it a combination of client list, and splash page or none of them  because people sign-up irrespective of logo, nav bar, splash page, or client list).

(Annotations in image  example below)


Caution: Typically when a large number of factors are under study, inefficient researchers abandon the ideas of factorial designs and revert to piece-wise experimentation. Using this approach, one control experiment is performed then additional experiments are performed by varying each factor to a new setting while holding all other factors constant to the control value. This is a poor strategy since the effect of each factor is determined by the difference in response between the two runs. The experimental noise may overshadow the factor effects and there is no averaging of data to reduce the noise.

Another strategy often used when a large number of factors are under study is to arbitrarily, or by opinion, pick a subset of the factors and do a complete factorial design. This is even a worse strategy, since the whole purpose of working with a large number of factors is to determine which subset of the factors are most important. By guessing, it is very likely that one or all of the most important factors will be left out of the design.


What is required to conduct a successful Multivariate Test?

Real-time randomization of traffic to ensure statistical validity and prevention of non-randomization bias. For example, if a plant is switched from one supplier of raw-material to another and notices a deterioration in yield one explanation is that the new raw material is worse than the other but another plausible scenario is normal drifting of the process due to other things changing gave a worse yield after the switch so the raw material source had nothing to do with the drop in yield. (A valid test in this case would be a random switch back and forth several times between suppliers and then test the outcome).


Who and What is Taguchi?

Genichi Taguchi was the person most responsible for developing high quality Japanese cars. He developed a technique to find the sources of variance and the method (Taguchi)  is named after him.

Taguchi Method is the approach popularized to find sources of variance. It leverages Taguchi arrays which are orthogonal arrays that minimize the number of experiments needed to gather information for analysis.

Taguchi Method

What are Factors?

Factors are elements that you want to test.

In the case of Amazon Kindle you would want to test the Banner Color, Kindle Color, Price Text size, and Button.


What are Levels?

Levels are options for each factor.

In the case of Amazon Kindle you could have the Banner color in (Lime, Blue, or Crimson), Kindle color in (Black, White, and Grey), Price Text Size in (18 pt or 20 pt), and “Buy Now” button vs. “Order” button.


What are Orthogonal Arrays?

Orthogonal arrays are matrices that show which elements to include in an experimental run.

Run 16 => A2, B3, C1

Factor A Level 2, Factor B Level 3, Factor C Level 1


What is Full vs. Fractional Factorial?

In Full Factorial an experiment is run for every combination of levels. Example: 3 * 3 * 3 = 27 runs.

In Fractional Factorial an experiment is run for only a fraction of all possible combinations.


Comparison 9 runs vs. 27

Net, net, here are the 7 steps to complete your MVT journey

1. Determine measure you want to optimize. (Hypothesis generation)

2. Come up with possible changes.

3. Use number of changes (Factors) and options for each change (Levels) to detemine which runs/configurations are necessary.

4. Create a page for each experimental run.

5. Ensure that you minimize the non randomization bias by ensuring traffic is randomly sent to each of the test pages.

6. Run Tests.

7. Measure results in your favorite platform.

MVT Results Google Optimizer

(Source: Google Optimizer)


That’s it when it comes to nuts and bolts of MVT.  In a follow-up post I will share a live example from a site to conduct a potential Multivariate Test.

Contributed by Kanishka, originally posted at http://webpersuasion.com/web-analytics/primer-multivariate-testing