Hi everyone once you are comfortable with spark.One of the issues faced with running spark application is debugging its failure.

The simplest way to monitor spark jobs is Spark UI , However the problem with spark UI is that in case of completed application,It is difficult to understand the exact reason for failure.

What if I say we can customize the spark application and we can see the spark metadata attributes from app to task level for each and every line we write in spark code which can come very much handy when debugging application by just adding one line in spark submit

SparkListener Interface:

It is one of the developer api interfaces provided by spark which we can leverage by extending the interface for better understanding of our application. In this tutorial for better logging I have used the below methods so that I can visualize what is happening for each and every line I write with spark

1. onJobStart — What needs to be triggered when a job start
2. onJobEnd -What needs to be triggered when a job start
3. onApplicationStart -What needs to be triggered when a app start
4. onApplicationEnd -What needs to be triggered when a app end
5. onStageSubmitted — what needs to be trigger on stage is submitted
6. onStageCompleted- What needs to be trigger on stage end

How to use Spark Listener?

Step1: We can use spark listener by extending spark listener and overriding the default implementation as shown below

#pyspark #monitoring-and-evaluation #spark