We live in the era of big data. With the growth of the internet, data is rapidly growing in a huge amount and with high variability. Big data processing could give you a headache because it naturally takes a lot of running time. Apache Spark (or Spark) is one of the popular tools to process big data.

Spark is a unified analytics engine for large-scale data processing. With Spark, we can perform data processing quickly and distribute processing tasks across multiple computers. People use Spark because it is deployable in popular programming languages such as Python, Scala, Java, R and SQL. It also has a stack of libraries that support streaming data, machine learning and graph processing.

One of the Spark interfaces is PySpark that allows you to write Spark applications using Python APIs. PySpark supports Spark’s features such as Spark SQL, DataFrame, Streaming, MLlib (Machine Learning) and Spark Core. As someone working a lot with Pandas, I found that PySpark can do what I do there. However, the implementation is quite different. I will list some of the data processing tasks I usually perform in Pandas and translate them to PySpark.

#pandas #big-data-analytics #data-engineering #pyspark #spark #pandas data processing tasks translated to pyspark