Quick Tip: Using Git With NiFi Registry in Docker
Apache's NiFi Registry with Git seems like a match made in heaven, but it isn't without its quirks. This post will help you navigate them.
Join the DZone community and get the full member experience.Join For Free
Apache NiFi is a great tool for handling data flows, however, the flow development lifecycle has been slightly challenging.
The recent release of NiFi Registry, a sub-project to provide shared resources across instances of NiFi, initially provides the capability to manage versioned flows. As of version 0.2.0, NiFi Registry added support for persisting flow snapshots to Git, making it very compelling!
In this post, we’ll see how to set this up for use when developing NiFi flows in a dockerized environment.
There are a couple of tricks to get this working:
- Using an external volume for the flow snapshots (initialized as a Git repo or NiFi Registry will fail to start)
- Externalising the configuration for GitFlowPersistenceProvider
Setting Up NiFi Registry in Docker
- A working Docker installation
- NiFi 1.5+ (necessary for NiFi Registry integration)
Note that operating system commands (*nix-based) appear as follows:
Getting the Container Image
The Apache NiFi project publish NiFi Registry docker container images on Docker Hub. Let’s pull this:
docker pull apache/nifi-registry:0.2.0
Create a Directory for the Config
Subsequent steps assume that your current working directory is as follows:
mkdir nifi-registry; cd nifi-registry
Create the Flow Snapshots Directory
This will be used as a volume by the container:
mkdir flow_storage; pushd flow_storage
It must be initialized before use otherwise NiFi Registry won’t start and the container will be terminated.
This can be initialized using either git init or by cloning an existing repo (see the admin guide). For simplicity, we’ll execute the git init command:
Git Flow Persistence Configuration
The default nifi-registry.properties points to ./conf/providers.xml, so we will mount this file from outside the container.
You’ll need to create providers.xml with the following content in the current working directory (e.g. nifi-registry):
Paste this XML (and then save the file), which configures the GitFlowPersistenceProvider to use the externalized volume:
<providers> <flowPersistenceProvider> <class>org.apache.nifi.registry.provider.flow.git.GitFlowPersistenceProvider</class> <property name="Flow Storage Directory">/flow_storage</property> </flowPersistenceProvider> </providers>
There are more optional configuration options, such as the ability to push from NiFi – see the configuration for GitFlowPersistenceProvider in the admin guide.
Create the Container
docker run --name nifi-registry -p 18080:18080 -d -v $PWD/flow_storage:/flow_storage -v $PWD/providers.xml:/opt/nifi- registry/nifi-registry-0.2.0/conf/providers.xml apache/nifi- registry:0.2.0
Create a Bucket
NiFi uses buckets within the registry to store flows, so it makes sense to create one first. This is achieved using the NiFi Registry administration UI, available on http://localhost:18080/nifi-registry.
Bucket creation is available under settings (see this section of the getting started guide).
Connect NiFi to the NiFi Registry
The documentation shows how to connect NiFi to the NiFi Registry.
If you are running both NiFi and NiFi Registry in Docker containers, then you will need to use the IP address of the host machine and the exposed port number. The host IP address can be seen as the Gateway address in the output of
docker inspect nifi-registry (e.g. 172.17.0.1).
If you’ve followed this quick tip through, then you should now have the ability to version flows, commit them to git and push them (external to NiFi).
Published at DZone with permission of Robin Bramley, DZone MVB. See the original article here.
Opinions expressed by DZone contributors are their own.