A Primer on Multivariate Testing

In my last post on testing I received some great feedback from my readers. One such feedback was around “describing” the principles of MVT. The question was “It would be great if you would be able to get into further detail on the principles behind this (MVT) form of testing”. This question has become the motivation behind this persuasion. In this post  we will recap  the Why’s of testing and experimentation and then jump into the principles of Multivariate Testing and how it impacts business performance. We will also look at designing experiments with Multivariate Testing (Levels, Factors, and Orthogonal Arrays).

Sounds complicated? Don’t worry. The goal of this persuasion is to provide you with a straight forward description of Multivariate Testing so you can walk the walk and talk the talk in real world scenarios. If you stick with me,  I am sure  you will elevate your learning around MVT.

It’s worth pointing out that proactive testing and experimentation offers a sustainable competitive advantage. Majority of the digital properties evolve in response to their competition and that evolution is slow (and poor). Testing and Experimentation serves as an enabler to “bring the organization along” to:

Complement the “art” of web design (right brain) with “science” (left brain) by validating design decisions with statistically significant findings by identifying factors influencing your business objectives. More importantly controlled experiments can help you quantify the variables which have a significant effect on your desired response by screening out analysis of insignificant variables. (Correlation is not Causation)

Optimize site improvements at any stage  of the organizational evolution. If you are new to testing, you can hit the ground running by adopting simple frameworks; if you are already performing some testing you can turbo charge your improvement process by leveraging principles of successful experimental design techniques illustrated in this persuasion.


Less is More


Magnify the impact of Web Analytics. This is my personal favorite as it enables your group to be strategic (vs. ad hoc) and empowers you to accelerate the evolution of your digital presence through proactive discovery.



How do you define Multivariate Testing and What are it’s benefits?

This form of testing involves studying the effects of multiple factors simultaneously  in an efficient way so as to isolate the most important elements and combinations on a desired outcome. I discussed this at a high level in my prior post. This form of testing is ideally suited to Web Analytics (more than any other industry).

It also yields a significantly better set of results with fraction of the effort required by other methods.

Finally, the results for this form of testing are almost immediate with neglible costs when compared to other testing approaches.

…. and its advantages?

Efficiency: By significantly fewer testing iterations than A/B one can take less time and perform test and learn of key variables that drive conversion with minimal cost.

Optimizing Combination of Variables: Allowing for testing of possible interaction effects between variables.

As an example, here is how you can consider performing a Multivariate Test on Google Analytics homepage. In this case user can potentially test Position of Logo, Navigation Bar Type, Splash Page, and Selection of Clients to drive their desired macro (or micro) outcomes.

For example, Google may want to know which of the factors/levels (see description below) contributes to signing-up for their Analytics platform. (Is it a combination of client list, and splash page or none of them  because people sign-up irrespective of logo, nav bar, splash page, or client list).

(Annotations in image  example below)


Caution: Typically when a large number of factors are under study, inefficient researchers abandon the ideas of factorial designs and revert to piece-wise experimentation. Using this approach, one control experiment is performed then additional experiments are performed by varying each factor to a new setting while holding all other factors constant to the control value. This is a poor strategy since the effect of each factor is determined by the difference in response between the two runs. The experimental noise may overshadow the factor effects and there is no averaging of data to reduce the noise.

Another strategy often used when a large number of factors are under study is to arbitrarily, or by opinion, pick a subset of the factors and do a complete factorial design. This is even a worse strategy, since the whole purpose of working with a large number of factors is to determine which subset of the factors are most important. By guessing, it is very likely that one or all of the most important factors will be left out of the design.


What is required to conduct a successful Multivariate Test?

Real-time randomization of traffic to ensure statistical validity and prevention of non-randomization bias. For example, if a plant is switched from one supplier of raw-material to another and notices a deterioration in yield one explanation is that the new raw material is worse than the other but another plausible scenario is normal drifting of the process due to other things changing gave a worse yield after the switch so the raw material source had nothing to do with the drop in yield. (A valid test in this case would be a random switch back and forth several times between suppliers and then test the outcome).


Who and What is Taguchi?

Genichi Taguchi was the person most responsible for developing high quality Japanese cars. He developed a technique to find the sources of variance and the method (Taguchi)  is named after him.

Taguchi Method is the approach popularized to find sources of variance. It leverages Taguchi arrays which are orthogonal arrays that minimize the number of experiments needed to gather information for analysis.

Taguchi Method

What are Factors?

Factors are elements that you want to test.

In the case of Amazon Kindle you would want to test the Banner Color, Kindle Color, Price Text size, and Button.


What are Levels?

Levels are options for each factor.

In the case of Amazon Kindle you could have the Banner color in (Lime, Blue, or Crimson), Kindle color in (Black, White, and Grey), Price Text Size in (18 pt or 20 pt), and “Buy Now” button vs. “Order” button.


What are Orthogonal Arrays?

Orthogonal arrays are matrices that show which elements to include in an experimental run.

Run 16 => A2, B3, C1

Factor A Level 2, Factor B Level 3, Factor C Level 1


What is Full vs. Fractional Factorial?

In Full Factorial an experiment is run for every combination of levels. Example: 3 * 3 * 3 = 27 runs.

In Fractional Factorial an experiment is run for only a fraction of all possible combinations.


Comparison 9 runs vs. 27

Net, net, here are the 7 steps to complete your MVT journey

1. Determine measure you want to optimize. (Hypothesis generation)

2. Come up with possible changes.

3. Use number of changes (Factors) and options for each change (Levels) to detemine which runs/configurations are necessary.

4. Create a page for each experimental run.

5. Ensure that you minimize the non randomization bias by ensuring traffic is randomly sent to each of the test pages.

6. Run Tests.

7. Measure results in your favorite platform.

MVT Results Google Optimizer

(Source: Google Optimizer)


That’s it when it comes to nuts and bolts of MVT.  In a follow-up post I will share a live example from a site to conduct a potential Multivariate Test.

Contributed by Kanishka, originally posted at http://webpersuasion.com/web-analytics/primer-multivariate-testing

2 thoughts on “A Primer on Multivariate Testing

  1. Thanks for this. I think it makes sense to break MVT into two separate, but related activities 1) Data collection; 2) Data analysis.
    The data collection bit is what you are testing and experiment setup (full factorial, fractional, segmented, etc.). Most of this post is about how to collect the data.
    The second is the data analysis portion. For MVT it is a little more than just picking the best performing combination a’ la the pairwise approach in A/B testing. This is esp true if the test is fractional, since at the front end you have excluded possible combinations.
    One estimates the main effects (and optionally the interaction effects up to a certain order) of each variable in order to find the ‘best’ combination.
    An additional benefit of MVT is hat you can tease out the impact of each of the variables and determine how much each affects the total variance of the results (importance). Which can be useful to help guide future test/changes as well as getting at marginal value of each variable wrt. your success measure.
    Perhaps a small discussion of the factorial ANOVA would help illustrate the analysis side of MVT.
    Thanks again for taking the time to post this.

  2. Matt:
    Thanks very much for your comment. You have inspired me to take this topic as a solution centric approach as a follow-up post.
    Analysis Of Variance is a methodology consisting of a collection of statistical models for experimental data that divides the observed variations into different parts. T-Tests as you rightly point out are useful in determining the significance between two group means. Anova instead uses F-test followed by F-Table to determine if there is a significant difference in group means.
    I am preaching to the choir but I wanted to acknowledge your comment and I will incorporate it in the next post.

Leave a Reply

Your email address will not be published. Required fields are marked *

5 Skills Every Digital Marketer Needs to Succeed – Free eBook





Download a Free eBook that can lead you to a path full of success!