How many of you use p=0.05 as an absolute cut off? p ≥ 0.05 means not significant. No evidence. Nada. And then p < 0.05 great it’s significant. This is a crude way of using p-values, and hopefully I will convince you of this.

What is a p-value?

A lot of us use p-values following this arbitrary cut off but don’t actually know the theoretical background of a p-value. A p-value is the probability, under the null hypothesis, of observing data at least as extreme as the observed data. It is not, for example, the probability that some population parameter x = 0. x either equals 0 or it does not (in a frequentist setting).

So, the smaller the p-value, the more unlikely it is that this data would have been observed under the null hypothesis. In essence, the smaller the p-value, the stronger the evidence against the null hypothesis.

What affects p-values?

Two things mainly. The first is the strength of effect. The greater the difference from the null hypothesis. The smaller the p-value will be.

The second is the sample size. The larger the sample, the smaller the p-value will be (if in fact the null hypothesis is false).

So, this means that if p ≥ 0.05, it could be because the effect isn’t that strong (or doesn’t exist) or that our sample is too small, resulting in our test being underpowered to detect a difference.

#p-value #mathematics #statistics #data-science #data analysis