The process of benchmarking is considered to be one of the most crucial assets for the progress of AI and machine learning research. The benchmark datasets are usually fixed sets of data, which are manually, semi-automatically as well as automatically generated to form a representative sample for these specific tasks to be solved by a model.

Recently, researchers from the Institute for Artificial Intelligence and Decision Support, Vienna claimed that the considerable part of metrics currently used to evaluate classification AI benchmark tasks might be inconsistent. It may result in a poor reflection in the performance of a classifier, especially when used with imbalanced datasets.

For the research, they analysed the present aspect of performance metrics that are based on data covering more than 3500 ML model performance results from a web-based open platform.

#developers corner #ai research benchmark #ai research papers #benchmark #benchmarking ai #bias in ml research #inconsistent benchmark