Yarn CheatSheet

Run sample Yarn application

Distributed Shell Program

yarn org.apache.hadoop.yarn.applications.distributedshell.Client -shell_command ls -num_containers 1 -jar /usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-distributedshell.jar -timeout 300000 --queue default

Simple Pi Job

yarn jar /usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-examples.jar pi 1 1
yarn jar /usr/hdp/current/hadoop-mapreduce/hadoop-mapreduce-examples.jar pi 8 8

To gather application logs

yarn logs -applicationId <appId>

To gather any specific container logs

yarn logs -applicationId <appId> -containerId <conatinerId>

To get the status of the queue

yarn queue -status llap 
yarn queue -status default 

To gather application master logs

yarn logs -applicationId <Application ID> -am ALL

Spark Shell Command

spark-shell --executor-memory 5G --executor-cores 1 --num-executors 1 --driver-memory 5G --queue Batch

Leave a Comment