Pyspark is a Python API that supports Apache Spark, a distributed framework made for handling big data analysis. It’s an amazing framework to use when you are working with huge datasets, and it’s becoming a must-have skill for any data scientist.

In this tutorial, I will present how to use Pyspark to do exactly what you are used to see in a Kaggle notebook (cleaning, EDA, feature engineering and building models).

I used a database containing information about customers for a telecom company. The objective is to predict which clients will leave (Churn) in the upcoming three months. The CSV file with the data contains more than 800,000 rows and 8 features, as well as a binary Churn variable.

#machine-learning #apache-spark #big-data #pyspark #python