How to launch spark-shell in debug mode

      No Comments on How to launch spark-shell in debug mode

Most of the time to troubleshoot issues with spark-shell we need to enable the debug mode so we can print more information on the screen to understand the issue. Debug log prints all information and it helps to resolve the issue. Today we will discuss how to enable the debug on the spark-shell

First, we will create a custom log4j file

cat /tmp/log4j.properties
 log4j.rootCategory=debug,console
 log4j.logger.com.demo.package=debug,console
 log4j.additivity.com.demo.package=false
 log4j.appender.console=org.apache.log4j.ConsoleAppender
 log4j.appender.console.target=System.out
 log4j.appender.console.immediateFlush=true
 log4j.appender.console.encoding=UTF-8
 log4j.appender.console.layout=org.apache.log4j.PatternLayout
 log4j.appender.console.layout.conversionPattern=%d [%t] %-5p %c - %m%n

Then we can pass that file while launching the shell from the command line. To launch my spark-shell I will use below command

spark-shell --master yarn --deploy-mode client --files /tmp/log4j.properties --conf "spark.executor.extraJavaOptions='-Dlog4j.configuration=log4j.properties'" --driver-java-options "-Dlog4j.configuration=file:/tmp/log4j.properties"

Once you ran the spark shell command using above command you will see debug logs being printed on the screen. This will help you resolve any issues with the configuration of spark

Leave a Reply