How to launch spark-shell in debug mode

Most of the time to troubleshoot issues with spark-shell we need to enable the debug mode so we can print more information on the screen to understand the issue. Debug log prints all information and it helps to resolve the issue. Today we will discuss how to enable the debug on the spark-shell

First, we will create a custom log4j file

cat /tmp/
 log4j.appender.console.layout.conversionPattern=%d [%t] %-5p %c - %m%n

Then we can pass that file while launching the shell from the command line. To launch my spark-shell I will use below command

spark-shell --master yarn --deploy-mode client --files /tmp/ --conf "spark.executor.extraJavaOptions=''" --driver-java-options "-Dlog4j.configuration=file:/tmp/"

Once you ran the spark shell command using above command you will see debug logs being printed on the screen. This will help you resolve any issues with the configuration of spark

Leave a Comment