What is Imbalanced Data

One of the most common problems when working with classification tasks is imbalanced data where one class is dominating over the other. For example, in the Credit Card fraud detection task, there will be very few fraud transactions (positive class) when compared with non-fraud transactions (negative class). Sometimes, it is even possible that 99.99% of transactions will be non-fraud and only 0.01% of transactions will be fraud transactions.

You can have a class imbalance problem on binary classification tasks as well as multi-class classification tasks. However, the techniques we are going to learn here can be applied to both.

Why should you worry about Imbalanced Data?

Consider the same example of credit card fraud transaction detection where fraud and non-fraud transactions are in the ratio of 99% and 1% respectively. This is a highly imbalanced dataset. If you were to train the model on this dataset, you will get accuracy as high as 99% because the classifier will pick up the patterns in the popular classes and predict almost everything as non-fraud transactions. As a result, the model will fail to generalize on the new data. This is also the reason why accuracy is not a good evaluation metric when dealing with imbalanced data.

#data-science #machine-learning #classification #imbalanced-data #data