A look at how personal bias, statistical pitfalls, and intellectual dishonesty can affect data analysis

Over the past decade, powerful new tools have empowered organizations to gather and analyze data at much lower cost—enabling previously unimaginable power when it comes to predicting outcomes and behaviors. Given the availability of granular data and raw computing power accessible to us, data professionals are able to create powerful models that can predict patterns of consumption, travel, and behavior with ever-increasing accuracy. Ethical implications of this are many—far more complicated and nuanced than the scope of this article can cover. This article seeks to address the subject of honest engagement with data analysis.

Thanks to the nexus of availability and computability of data, institutions are turning to analysis and regression to solve a larger share of problems and answer a larger share of questions. This places responsibility on the shoulders of those who work with data to provide meaningful, but more importantly, accurate and unbiased analysis to stakeholders.

To meet this standard, a data professional needs to avoid using unrigorous statistical methods and inaccurately modeling, yes—but they also need to be statistically/intellectually honest. There are many avenues through which a data professional can avoid being dishonest—either intentionally or unintentionally—being aware of them can help avoid drawing fallacious conclusions.

#analytics #data #information-technology #statistics #big data comes big responsibility #big responsibility