The topic of this series is something that appears sparingly in standard ML courses but frequently in the industry — probability calibration. In the first post of the series, I will give a general introduction to probability calibration for classifiers and discuss when it makes sense to use calibrators.

Introduction

In machine learning, sometimes probabilistic classifiers are needed — classifiers that not only return the most likely class label, but also the probability of that class. A probabilistic classifier is well-calibrated when the predicted probability matches the true probability of the event of interest. For example, if a fraud classifier returns 0.1, or 10%, for the likelihood of a particular credit card application to be fraudulent, this number is considered well-calibrated if similar types of applications are truly fraudulent on average in 1 of 10 samples.

This is important when the absolute value of the predicted probability, instead of just the rank order, matters for the modeler. An example of this is sports betting.

Before the 2018 World Cup, VegasInsider.com posted 5/1 odds of Germany winning the tournament, which means for every dollar, you get $6 back (original dollar plus $5 payout) if Germany wins and $0 otherwise. So if you want to bet on Germany, you’d better be sure that Germany has more than ⅙ chance of winning. If you built a classifier to predict the winning probability of each team, and its output is Germany: 0.25 and England: 0.1, you want to make sure that classifier is telling you Germany has a 25% chance of winning, instead of merely saying Germany has a better chance than England.

#probability #maching-learning #calibration