Photo by Jeswin Thomas / Unsplash

Starting My First A/B Testing Using Optimizely


I’ve been playing around with Optimizely to run A/B and multivariate testing. I’d thought I share my experience around this optimization efforts.

I came across Optimizely during eMetrics and Conversion summit in San Francisco. I happened to be in the breakout session where Optimizely founder Dan Siroker was presenting. He was the Director of Analytics for the Obama Presidential Campaign and Deputy New Media Director for the Presidential Transition. In his presentation, he went through the slides showing the learnings on various tests he ran for Obama’s fund raising.

Anyways, what’s interesting is that the tool allows you to run tests with minimal (or no) involvement of web engineer. Basic test experiments could be activated by putting a simple javascript on the page. The test setup could be done in their WYSIWYG interface where you can change, move, remove parts of the web page on the fly.

Optimizely Features
Optimizely edit elements

Custom rules on different instances to run test could simply do by leveraging the URL parameters, or custom variables defined in the javascript.

The test I set up was a little tricky. The same exact URL is used for two different page because the state of the page is determined by some dynamic logic within the code. I overcame that by defining a custom variable in javascript.

To make sure the proper page is being tested, I added additional parameter and value to make sure it is not impacting similar URL elsewhere.

Another challenge that I encountered was the swapping dynamic element. For example, some images like Trust-e or Geo Trust logos are generated dynamically. So when you use the “move” feature on Optimizely, it doesn’t move the entire logical element, so the page gets distorted when you push the test.

The customer support from Optimizely team is really fast, technical, and friendly. It was actually pretty surprising given that I am using the 200,000 visitor plan, but getting the service level like an enterprise customer. I hope they don’t change that aspect of the service, it really helps to have a quick and accurate turn around on issues that could impact the bottom-line.

I have to say the setting up the multivariate piece of the tool is a little clunky, given my experience from using Webtrends Optimize. However, the price and service Optimizely are just so compelling it really doesn’t matter. What matters is really churning out the test and get back the results, and move on to drive the lift in conversions.

What I learned from few tests is that it goes back to the basic fundamental thing.

Plan your testing wisely.

Because you can quickly set up tests and run little tests here and there, I felt that I was missing the big opportunities to get huge ROI. In my view, good planning would mean to do your planning due diligence:

  • Use your analytics tool measure where the low hanging fruit is in terms of consumer pain points. Find high traffic page with high bounces, traffic from a campaign where landing page is not effective, etc.
  • Create hypothesis on why the page is not effective. Ask yourself why does that page exist, what does that page need to do
  • A/B testing or MVT. They are two different tactics and require a different amount of resource/efforts to deliver. So think and plan wisely.
  • Set the right expectation with business partners/clients that getting results may take time to get accurate and significant read
  • Define your conversion and make sure key stakeholders know what you’re optimizing for

Seriously, these things are really important. Optimizely allows you to run ad-hoc tests really quickly, but it could also disappoint you if you’re not getting the results as expected after waiting few days or maybe weeks.

With Google Analytics integrating their Google Website Optimizer, and companies like Visual Website Optimizer (similar to Optimizely) emerging, the barrier to doing website optimization is becoming lower and less difficult.

I would say lastly, don’t be afraid and just go testing and learn quickly from your mistakes.

Thank you so much for taking the time to read my content.  If you've liked what I've had to say please subscribe!

Marketing Strategy

Kris Twitter

As a data journalist, I enjoy curating and analyzing marketing trends, and data. The things that fascinate me the most are the transforming business landscape due to evolving marketing technologies.