A focused study on the speed comparison of reading parquet files using PyArrow vs. reading identical CSV files with Pandas

Why Parquet in lieu of CSV?

Because you may want to read large data files 50X faster than what you can do with built-in functions of Pandas!

Comma-separated values (CSV) is a flat-file format used widely in data analytics. It is simple to work with and performs decently in small to medium data regimes. However, as you do data processing with bigger files (and also, perhaps, pay for the cloud-based storage of them), there are some excellent reasons to move towards file formats using the columnar data storage principle.

Apache Parquet is one of the most popular of these types. The article below discusses some of these advantages (as opposed to using the traditional row-based formats e.g. a flat CSV file).

#technology #python #big-data #data-science #how fast is reading parquet file (with arrow) vs. csv with pandas #parquet file