Apache Spark CheatSheet

      No Comments on Apache Spark CheatSheet

To get all spark configs from spark shell

sc._conf.getAll()

To give specific GC overhead memory

--conf spark.yarn.executor.memoryOverhead=12288 

To enable GC logging

--conf "spark.driver.extraJavaOptions=-verbose:gc -XX:+PrintGCDetails" --conf "spark.executor.extraJavaOptions=-verbose:gc -XX:+PrintGCDetails" 

Leave a Reply