Probability Distributions:

Probability distributions are fundamental to statistics, just like data structures are to computer science.

Things happen all the time:

1. dice are rolled,

2. it rains,

3. buses arrive.

After the fact, the specific outcomes are certain:

• the dice came up 3 and 4,
• there was half an inch of rain today,
• the bus took 3 minutes to arrive.

Probability distributions describe what we think the probability of each outcome is, which is sometimes more interesting to know than simply which single outcome is most likely.

They come in many shapes, but in only one size: probabilities in a distribution always add up to 1.

For example, flipping a fair coin has two outcomes: it lands heads or tails.

Before the flip, we believe there’s a 1 in 2 chance, or 0.5 probability, of heads. The same is true for tails.

That’s a probability distribution over the two outcomes of the flip, and this is an example of Bernoulli distribution.

Above, both outcomes were equally likely, and that’s what’s illustrated in the diagram. The Bernoulli PDF has two lines of equal height, representing the two equally-probable outcomes of 0 and 1 at either end.

Exponential Family

Some of the basic linear regression and classification algorithms can also be derived from the general form of Exponential Family.

A distribution belongs to exponential family if it can be transformed into the general form:

where

η is canonical parameter

T(x) is sufficient statistic

A(η) is cumulant function or log partition function

Exponential family includes the Gaussian, binomial, multinomial, Poisson, Gamma and many others distributions.

A fixed choice of T, a and h defines a family (or set) of distributions that is parameterized by η; as we vary η, we then get different distributions within this family.

Some examples of Exponential Family Distributions used in Machine learning

1. Bernoulli Distribution

2. Gaussian Distribution

etc…

#machine-learning #linear-models #glm #artificial-intelligence #basics-machine-learning