Long before I took any statistical class, I’ve heard that A/B testing is almost a must for data analyst interviews. So I Googled it and thought: Hmm…isn’t it just like the control and experiment studies we conducted in high school biology class? Simple as it may sound, it actually has a rigorous statistical process and involves various business concerns.

What is A/B testing?

A/B testing is a way to compare two versions of a_ single variable__ , typically by testing a subject’s response to variant A against variant B , and determining _which of the two variants is more effective _[2]._

For a better understanding, I will use this example throughout this article to give a more concrete explanation:

Suppose an online learning platform wants you to test whether they should change the web page’s button size to increase the number of users.

Single variable:_ “Join Now ” button size on a webpage_

Variant A:_ 4 x 3 button size_

Variant B:_ 16 x 9 button size_

_Subject’s response: _click-through probability changes

Goal:_ find out which option has a higher click-through probability_

Jack of all trades?

Sounds like if we change the variables and responses to any other attributes, A/B testing can still apply, huh? Indeed, A/B testing has many Use Cases, including:

UI changes, recommendations, ranking changes, implicit changes such as loading time, etc.

However, there are also cases it is Not So Useful:

• Missing items
• For our online course website example: if there are any courses we didn’t offer, but the users are looking for, A/B testing cannot tell.
• New experiences
• Introducing new experiences such as VIP services can be troublesome because:

a) The baseline of comparison is not clear

b) The time needed for users to adapt to new experiments can be quite costly, as there might be some psychological influences on users:

Change Aversion: when faced with a new interface or changed functionality, users often experience anxiety and confusion, resulting in a short-term negative effect.

Novelty Effect: when new techs came out, users will often have increased interests so that the performance will improve initially, but it’s not because of any actual improvement.

#statistics #data-science #data-analytics #data-analysis