A/B Testing: Observe And Report

Originally posted for DMXENGAGE on April 9

In the first part of our scientific approach to A/B testing, we discussed the importance of determining what to test and how to execute your A/B test experiment. Some novice scientists (marketers) assume that the work is over after the experiment is completed. And while it is true that the majority of the heavy lifting is over, your A/B journey is just getting started.

Observe and Record

Once you complete your experiment, it’s time to pull out your clipboard and record your results. Thanks to all of the amazing marketing technology we have at our fingertips these days, this part of the marketing scientific method is easier than ever. The marketing automation platform that you are using records opens/clicks, responses, form submissions, etc. At the end of your A/B test time frame, analyze your results.

After gathering the results of the West Point Envelope test, our team was surprised by the results:


Full-Color, Graphic Envelope: 3.3% response rate

Plain, White Envelope: 3.6% response rate


Think back to your hypothesis. Can you confirm or deny your hypothesis? After reviewing the results, we determined that our hypothesis was incorrect. The full-color, graphic envelope did not generate a higher response rate. Now, it doesn’t matter if you win or lose, but how you play the game, right? So take some time to think aboutwhy your hypothesis was incorrect (or correct!)

So why did the plain, white email generate a high response rate? Perhaps it’s because it looked more official and important, whereas the full-color envelope looked like marketing material? Or perhaps the student was simply more curious about what was in the plain envelope? The analysis portion of your test will help you determine more A/B test hypotheses.

Another important part of the analysis step is to determine statistical significance. Don’t panic, there are calculators for that too. (If you want go deep into understanding statistical significance, we wish you the best of luck and offer this helpful link.) But it is important to determine the statistical significance to account for any sampling testing errors. Our test was a virtual tie, making the result not statistically significant. However, it did give us enough data to return to our client and explain that the ROI of a fancy, full-color envelope may not be worth the cost.

Report Results

Finally, report your results. Whether it is to your client, your boss or marketing mavens like yourselves, share your data! We marketing nerds want to know!

If you are a said marketing nerd, you can find the full story of our direct mail envelope test here.

Test Again

Your job as a scientific marketer is never over. Once you complete an A/B test, determine how you can adjust your hypothesis and test to gain more information about what is driving your customers to compete the required action.

Now that we know that the plain, white envelope has a slightly higher open rate, we can recommend other A/B tests to execute. These tests could focus on the direct mail letter, or the layout of the personalized landing page. The possibilities are endless and the marketing science fun is just getting started.

What have been some of your most surprising A/B test results? Loosen up your pocket protector and report your favorite scientific marketing nerd results in the comments.

Leave a Reply

Your email address will not be published. Required fields are marked *