When we start studying the concepts of probability and statistics, there are a few topics that require us to take a logical leap, often leaving us confused. In my earlier post, I have talked about one such topic, confidence interval. In this post, I will try to explain another such confusing topic, P-value. (Spoiler alert: No, it is not probability, but is related to probability)
After this post, you will be able to understand the correct interpretation of P-value and how the P-value leads to rejection/failure of rejection of hypothesis testing. Any mathematical equation or concept becomes beautiful if we learn how to read it in simple English and that is exactly what I am trying to do here.
I would assume that you are aware of the basics of hypothesis testing and conditional probability.
Conditional probability: P(it will rain today | sky is grey) read as probability that it will rain today given that sky is grey is 0.4 means that there is only a 4% chance that it will rain today if the sky has become grey.
Hypothesis testing: a brief overview
Real-world scenario
Imagine you are a data scientist at Expedia. You need to analyze what factors drive the new users to become loyal customers of Expedia. You do some initial exploratory analysis of the data. You find out that when a new user book through some promotional offer or deal, he/she tends to return to the website. So if a new user in his first visit uses some promotional offer to book a service in Expedia, they are more likely to come back than those first-time users who booked directly without using any deal.
Having studied a little about arts, you notice that these deals and offers are not eye-catching to the website visitors because those are displayed in blue color. You think that if the deals and offers are redesigned and highlighted in red color, this might increase the retention rate of first-time users. You recommend this to business. But rather than simple idea narration, to show how confident you are in your claim statistically. You perform the following steps:
- Redesign the promotional content of the website in red color, and show this new version only to half of the new traffic to the website
- The rest of the new users see the old version of the website, i.e., blue color.
If in a day 100,000 new users visit the website, imagine 50,000 sees the old version and rest of the 50,000 sees the new version
In the new version, Number of new users who used the promo and returned after their first use is 7700
In the old version, Number of new users who used the promo and returned after their first use is 7000
Here, you observe a 10% increase in the number of retained users when they used the new version of the website. But before you conclude that changing the website promo colors to red can increase the retention rate to 10%, you must be sure that this increase is not by any random chance. This increase might be due to the way you sampled or because in general users who used a new version that day do not know any other travel booking website or because of any other random reason. For this clarity, you run a hypothesis test.
Hypothesis testing methodology
- Choose one test statistic (retention percentage here)
- Formulate hypothesis
**_Null Hypothesis (Always status quo): _**There is no difference in the retention percentage of new users between the two versions. Retention percentage of new version = retention percentage of the old version. The difference observed is just random.
Alternative hypothesis: There is a difference between the retention percentage of the new version and the retention percentage of the old version.
#hypothesis-testing #data-science #statistics #p-value #machine-learning #deep learning
