NiFi Cheatsheet

Command to take thread dump

./bin/nifi.sh dump /tmp/thread-dump.txt

Command to take JStack

/usr/jdk64/jdk1.8.0_112/bin/jstat -gcutil <nifi pid> 5000

Enable debug on the particular processor

<logger name="org.apache.nifi.processors.standard.MergeContent" level="DEBUG"/>

To enable Trace on Extension Manager

<logger name="org.apache.nifi.processors.hadoop" level="TRACE"/>
<logger name="org.apache.nifi.nar.ExtensionManager" level="TRACE"/>

Enable debug on LDAP Provider

<logger name="org.apache.nifi.ldap.tenants.LdapUserGroupProvider" level="DEBUG"/>

General tuning for NiFi Properties

nifi.cluster.node.protocol.threads=70
nifi.cluster.node.protocol.max.threads=100
nifi.zookeeper.session.timeout=30 sec
nifi.zookeeper.connect.timeout=30 sec
nifi.cluster.node.connection.timeout=60 sec
nifi.cluster.node.read.timeout=60 sec
nifi.ui.autorefresh.interval=900 sec
nifi.cluster.protocol.heartbeat.interval=20 sec
nifi.components.status.repository.buffer.size=300
nifi.components.status.snapshot.frequency=5 mins
nifi.provenance.repository.implementation=org.apache.nifi.provenance.WriteAheadProvenanceRepository
Note: Makes sure they have commented out G1GC java.arg in NiFi’s bootstrap.conf file if they are switching to the WriteAhead Provenance implementation. Failure to do so can result in NiFi hitting Lucene bug in Java 8 resulting in corruption in NiFi’s repos.

General tuning at OS level

nifi  hard  nofile  999999  
nifi  soft   nofile  999999
nifi  hard  nproc 50000
nifi  soft   nproc 50000

Repair Corrupted Flowfile Repository

java -cp nifi-toolkit-flowfile-repo-1.10.0-SNAPSHOT-jar-with-dependencies.jar org.apache.nifi.toolkit.repos.flowfile.RepairCorruptedFileEndings corrupt-directory  new-directory
java -cp nifi-toolkit-flowfile-repo-1.10.0-SNAPSHOT-jar-with-dependencies.jar org.apache.nifi.toolkit.repos.flowfile.RemoveFlowFilesWithMissingContent corrupt-directory  new-directory

Supported features and versions

  • Ranger Group Based Policies Support – HDF 3.1.0 onwards – NiFi 1.5.0
  • LDAP/AD login username lower to upper – HDF 3.3.0 onwards – NiFi 1.8.0
  • Load balancing in connection – HDF 3.3. onwards – NiFi 1.8.0 Document
  • NiFi Toolkit does not pick up memory values Issue – Fixed after – HDF 3.1.2

Leave a Comment