Bobby 7:06 pm on Jun 11, 2012 Reply
Tags: a/b testing, ,   

Google Analytics now has the ability to do A/B testing using the Content Experiments feature. A methodology for using this feature already exists. You may have heard of it; it’s called The Scientific Method.

The Scientific Method

The Scientific Method can be broken down into 6 basic steps.

  1. Ask a Question
  2. Research
  3. Hypothesis
  4. Test
  5. Analyze
  6. Communicate

1. Ask a Question

All experiments start with a question. Here are a few questions to get you started in the realm of A/B testing:

  • Do visitors that view the testimonial video have a higher conversion rate?
  • Can we increase the donation amount if we use a different image in the call-to-action?
  • Can we get more registrations by simplifying the sign-up process?

2. Research

Before you put on the goggles and light the Bunsen burner, have a look at best practices for your industry and see if there are results for similar experiments already published. Your research may cause you to refine the question, ask a different question, or even spur ideas for variations.

3. Hypothesis

Content Experiments exist so the performance of websites can be improved. What is your prediction about how the variation(s) will perform against the original content? IMPORTANT: Make sure that you are tracking the metrics you need to answer your question.

4. Test

Start serving the variant(s) of the original content to visitors. Make sure that the original and variation(s) receive a statistically significant amount of traffic. Avoid drawing conclusions from experiments that are lacking enough data.

5. Analyze

Look at the metrics that are pertinent to your question. Look at your goals. What is the conversion rate for visitors that saw the video testimonials versus those who did not? What is the average donation amount for call-to-action image Y and how does that compare to image Z?

Also look at other goals and objectives that may have been affected by the experiment.

6. Communicate

Understand the results of the experiment and share conclusions with the marketing team. Take action and make content improvements based on Content Experiment findings.

Conversion Rate Is Not Everything

DO NOT just look at the metrics which answer the question. The Content Experiment pages do not exist in isolation, so do not analyze the results of the experiment in isolation. This is a system.

The objective of A/B testing is to get a better result. Look at the big picture; don’t just focus on the underlying hypothesis which prompted the experiments. Segment the test data. Are there any patterns among visitors? Take a holistic look at all the metrics. What are the unintended consequences of the experiment?

For example, in a recent engagement with a non-profit organization, we saw negligible conversion rate results for our A/B test. The difference was a conversion rate improvement of only 0.4%. However, when we looked at the experiment data holistically, we found that although the conversion rate was barely affected, the average donation amount was lifted 274%! If we stayed focused only on the metrics which answered “Which call-to-action has a higher conversion rate?” we would have completely lost out on this HUGE performance gain.

Conclusion

The Scientific method doesn’t apply only to testing content on your website. A/B testing can be effectively used in your other marketing channels as well. Try it in email marketing campaigns and social campaigns. After you’ve analyzed the data to see if your hypothesis is true, take a holistic look at the experiment before you determine what the next steps should be.

An important step that needs to be emphasized is Communicate Your Results! Why keep this information within your corner of the world? Get your insights out to all the teams that will benefit. Action needs be taken once an experiment is complete, or the entire exercise is purely scholastic. Content Experiment results are very actionable and lead to the optimization of your website.