Getting informative insights from the raw data in hand is vital in a successful machine learning project. The selection of the right machine learning algorithm and tuning of the model parameters to achieve better performance are possible only with proper data analytics in the pre-processing stage. Traditional statistical analysis is simple and powerful in extracting the essence out of the raw data.
Statistical analysis is performed reliably and quickly with statistical software packages. The famous multi-purpose language, Python, has a great collection of libraries and modules to do statistical analysis in a lucid way. In this article, we discuss a widely used statistical tool called ANOVA with hands-on Python codes.
ANOVA is one of the statistical tools that helps determine whether two or more data samples have significantly identical properties. Let’s assume a scenario- we have different samples collected independently from the same dataset for cross-validation. We wish to know whether the means of the collected samples are significantly the same. Another scenario- we have developed three different machine learning models. We have obtained a set of results, and we wish to know whether the models perform significantly in the same manner. Thus, there are many scenarios in practical applications where we may need to use ANOVA as part of data analytics.
ANOVA is the acronym for Analysis of Variance. It analyzes variations among different groups and within those groups of a dataset (technically termed as population). However, there are some assumptions that the data must hold to use ANOVA. They are as follows:
- The data follows normal distribution
- The variance of data is the same for all groups.
- Data among groups are independent of each other.
Math concept behind ANOVA and its usage can be explored with the following hands-on Python example.
#developers corner #analysis of variance #anova #data analytics #data preprocessing #post hoc #python #statistical significance #statistics #tukey
