In this article, I will show you how A/B Testing is carried out in a planned and controlled manner with a real-world example.
Identify where there is the most opportunity:
In any business, there is always room for experimentation and improvement. In fact, there will be plenty of such opportunities. Does that imply all those should be worked upon? Let me remind you that A/B Testing is time-consuming and has costs associated with it and hence a good ROI (Return on Investment) is a must to convince the higher management into this. For instance, if you are trying to increase the sales of your e-commerce site and for that, you are buying an ad-space (for say $1000) and see an increase of $250 in sales. Is that effort worth the spend and experiment? The next question would be ‘How do we decide or identify the right opportunity?’
Try starting from any one of these:
- The top-level goal: which is the prime objective of your organization such as the number of leads generated, revenue generated, etc.
- Dig into performance metrics: Investigate your operations and check how well the business has been performing recently across various functional areas and divisions. Start with the one which requires the most attention (need of the hour).
- **Do user research: **Gather user inputs and feedback on how they like the services provided and what kind of changes would make their experience better.
- Do market research: Check how your competitors are performing, their new products and services.
Know your Business and Customers.
Before you begin an experiment, be very sure about how your business currently performs. For instance, if you are running an e-commerce site, you need to have an understanding of the following metrics (not limited to).
- Website Traffic
- Traffic source
- Revenue per Traffic Source
- Average Order value
- Customer Lifetime Value
- New Customers and Returning Customers
- Conversion Rate etc.
Select your target metric:
While designing an experiment there could be 2 or more metrics that we would be interested in
- Metrics that are directly impacted by the experiment
- Metrics that are at the bottom of the funnel (our top-level goal)
Let us consider an example of making the category menu of our e-commerce site more prominent which would directly impact the number of interactions made by the user. However with increased engagement, one would expect a higher revenue. It’s always better to choose a metric that is directly impacted by the experiment, as choosing our top-line goal might lead to incorrect inferences or in fact, might need a longer duration to execute. In general, the various stages the customers go through on the site are treated as a funnel, and an example of it shown below.
Define your Target Audience:
While performing experimentation, it is highly essential for us to be more specific about our target audience. For instance, In our e-commerce site example, we might want to define the specific characteristics like geographical region, type of user (new or existing users), age group of users, etc. of the sample of visitors for the experiment to be more impactful.
State your hypothesis
A scientific hypothesis is just an idea or a proposal that can be tested scientifically. It is something more than a wild guess and less than an established truth. The Hypothesis or Idea must be very clear and concise for the experiment to be successful.
In our case, ‘Revamping the Category menu of our e-commerce site could lead to an increased conversion rate (Directly impacted metric)’ could be one way to start.
Just for the sake of better understanding our target metric selection, let us also test if ‘Revamping the Category Menu leads to increased revenue per customer (Our Top line goal)’
Testing Conversion Rate Hypothesis with Chi-Square Test:
As shown below, we check for statistical significance using Evan’s Awesome A/B Tools Chi-Square Test for the number of clicks made by users. Why the Chi-Square Test? Recalling from the previous article, since we are dealing with clicks made by the user which is either a click or no-click event (boolean outcome) we make use of it.
We see that there were a total of 300 clicks made by users from the control (sample 1)and 350 clicks made by users from treatment (sample 2)groups respectively. We see that with 95% confidence, the conversion rate of users from the treatment group is sufficiently higher (p=0.017). Ok, what is this p-value here? To know that, Just be a little more patient and keep reading!
#statistics #user-experience #hypothesis-testing #user-testing #ab-testing
