Yarn CheatSheet

      No Comments on Yarn CheatSheet

Run sample Yarn application Distributed Shell Program yarn org.apache.hadoop.yarn.applications.distributedshell.Client -shell_command ls -num_containers 1 -jar /usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-distributedshell.jar -timeout 300000 –queue default Simple Pi Job yarn jar /usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-examples.jar pi 1 1 yarn jar /usr/hdp/current/hadoop-mapreduce/hadoop-mapreduce-examples.jar pi 8 8 To gather application logs yarn logs -applicationId <appId> To gather any specific container logs yarn logs… Read more »

NiFi Cheatsheet

      No Comments on NiFi Cheatsheet

Command to take thread dump ./bin/nifi.sh dump /tmp/thread-dump.txt Command to take JStack /usr/jdk64/jdk1.8.0_112/bin/jstat -gcutil <nifi pid> 5000 Enable debug on the particular processor <logger name=”org.apache.nifi.processors.standard.MergeContent” level=”DEBUG”/> To enable Trace on Extension Manager <logger name=”org.apache.nifi.processors.hadoop” level=”TRACE”/> <logger name=”org.apache.nifi.nar.ExtensionManager” level=”TRACE”/> General tuning for NiFi Properties nifi.cluster.node.protocol.threads=70 nifi.cluster.node.protocol.max.threads=100 nifi.zookeeper.session.timeout=30 sec nifi.zookeeper.connect.timeout=30 sec nifi.cluster.node.connection.timeout=60… Read more »