How To Install Apache NiFi on CentOS 7

Apache NiFi is a popular data pipeline software that uses directed graphs for data routing. Using various processors available in NiFi you can design data pipeline and transform your data and can route it to various data streaming and storage tools.

One such example data pipeline in NiFi can tail your file on the system and we can send that file’s data to a Kafka topic. NiFi includes hundreds of processor which we can use to design our flow. NiFi presents us with one simple User Interface on which we can use a simple pick and drop functionality to draw our data pipelines.

In this tutorial, you will install Apache NiFi 1.11 on CentOS 7

Prerequisites

To follow along with you will need

  • One CentOS 7 server with root access
  • At least 4GB of RAM
  • OpenJDK 8 installed on your server. You can follow this tutorial for installation of JDK

Step 1 — Creating a User for NiFi

As you scale in NiFi flows you will need to set specific resources for NiFi process so it is necessary to create a non root user in CentOS to run your NiFi service.

useradd nifi -m

-m flag will ensure that useradd is creating home directory for nifi user

Step 2 — Increase Maximum File Handlers

As NiFi is IO-intensive application you will need to increase the open file handlers for nifi user. To increase that limit for nifi user we will edit file /etc/security/limits.conf and add below lines

vi /etc/security/limits.conf

Add below lines in the end of file limits.conf

nifi  hard  nofile  50000 
nifi  soft  nofile  50000

Step 3 — Increase Maximum Forked Processes

NiFi uses a pool of thread to run processor running on Canvas. According to configuration, NiFi may need to run high number of threads on Linux system so you will need to increase the maximum allowable number by increasing /etc/security/limits.conf

vi /etc/security/limits.conf

Add below line at the end of file limits.conf

nifi hard nproc 10000
nifi soft nproc 10000

Step 4 — Download and Extract NiFi Binaries

As a root user go inside nifi home directory in /home/nifi there download nifi latest binary using wget method

wget http://apachemirror.wuchna.com/nifi/1.11.0/nifi-1.11.0-bin.tar.gz

Once the tar is downloaded you can extract that tar file using

tar -xvf nifi-1.11.0-bin.tar.gz

rename folder nifi-1.11.0 to install so that our complete nifi install path will look like /home/nifi/install

mv nifi-1.11.0 install

Change the permission on all nifi install files to nifi user

chown nifi:nifi /home/nifi/install
cd /home/nifi/install
chown nifi:nifi * -R

Step 5 — Configure NiFi Server

By default, if we start NiFi service as root user it will use root user to run the service but as we want to user nifi user which we have created we will need to set that user as run.user for NiFi server

vi /home/nifi/install/conf/bootstrap.conf
run.as=nifi

To make NiFi perform better we will increase java heap size of NiFi server by increasing memory values for below parameters

java.arg.2=-Xms1024m
java.arg.3=-Xmx1024m

Step 6 — Start NiFi Server

To start NiFi service we can use script nifi.sh to start NiFi service

cd /home/nifi/install/bin
./nifi.sh start

Once service is started using above command you can verify NiFi service status using below command

./nifi.sh status

You will see output like this for running service

./nifi.sh status
 nifi.sh: JAVA_HOME not set; results may vary
 Java home:
 NiFi home: /home/nifi/install
 Bootstrap Config File: /home/nifi/install/conf/bootstrap.conf
 2020-02-02 07:48:28,124 INFO [main] org.apache.nifi.bootstrap.Command Apache NiFi is currently running, listening to Bootstrap on port 38576, PID=11158

Step 7 — Access NiFi Canvas UI

Once NiFi service is up and running you can access NiFi canvas using url

http://HOSTNAME:8080/nifi/

Leave a Comment