If you missed Part 1, you can check it out here.

Simple Apache NiFi Operations Dashboard - Part 2

To access data to display in our dashboard we will use some Spring Boot 2.06 Java 8 microservices to call Apache Hive 3.1.0 tables in HDP 3.0 on Hadoop 3.1.

We will have our website hosted and make REST Calls to Apache NiFi, our microservices, YARN, and other APIs.

As you can see we can easily incorporate data from HDP 3 — Apache Hive 3.1.0 in Spring Boot Java applications with not much trouble. You can see the Maven build script (all code is in GitHub).

Our motivation is to put all this data somewhere and show it on a dashboard that can use REST APIs for data access and updates. We may choose to use Apache NiFi for all REST APIs or we can do some in Apache NiFi. We are still exploring. We can also decide to change the backend to HBase 2.0, Phoenix, Druid or a combination of these. We will see.

Spring Boot 2.0.6 Loading

JSON Output

Spring Boot Microservices and UI

https://github.com/tspannhw/operations-dashboard

To start, I have a simple web page that calls one of the REST APIs.

The microservice can be run off of YARN 3.1, Kubernetes, CloudFoundry, OpenShift, or any machine that can run a simple Java 8 jar.

We can have this HTML as part of a larger dashboard or hosted anywhere.

For parsing the monitoring data we have some schemas for metrics, status, and bulletins.

Now that I'm monitoring data is in Apache Hive, I can query it with ease in Apache Zeppelin (or any JDBC/ODBC tool).

Apache Zeppelin Screens

We have a lot of reporting tasks for Monitoring NiFi.

We read from NiFi and send to NiFi, would be nice to have a dedicated reporting cluster.

Just Show Me Bulletins for MonitorMemory (You Can See That in Reporting Tasks)

NiFi Query To Limit Which Bulletins We Are Storing In Hive (For Now Just grab Errors)

Spring Boot Code for REST APIs

Metrics REST API Results

Bulletin REST API Results

Metrics Home Page

Run the Microservice

java -Xms512m -Xmx2048m -Dhdp.version=3.0.0 -Djava.net.preferIPv4Stack=true -jar target/operations-0.0.1-SNAPSHOT.jar

Maven POM

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.dataflowdeveloper</groupId>
<artifactId>operations</artifactId>
<version>0.0.1-SNAPSHOT</version>
<packaging>jar</packaging>
<name>operations</name>
<description>Apache Hive Operations Spring Boot</description>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.0.5.RELEASE</version>
<relativePath/>
</parent>


<properties>
<java.version>1.8</java.version>
</properties>


<dependencies>


<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-web</artifactId>
    <exclusions>
        <exclusion>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-logging</artifactId>
        </exclusion>
<exclusion>
 <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-tomcat</artifactId>
 </exclusion>
    </exclusions>
</dependency>
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-log4j2</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-jdbc</artifactId>
</dependency>
<dependency>
<groupId>org.apache.hive</groupId>
<artifactId>hive-jdbc</artifactId>
<version>3.1.0</version>
<exclusions>
<exclusion>
<groupId>org.eclipse.jetty.aggregate</groupId>
<artifactId>*</artifactId>
</exclusion>
<exclusion>
<artifactId>servlet-api</artifactId>
<groupId>javax.servlet</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework.restdocs</groupId>
<artifactId>spring-restdocs-mockmvc</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
</plugins>
</build>
   <repositories>
        <repository>
            <id>spring-releases</id>
            <url>https://repo.spring.io/libs-release</url>
        </repository>
    </repositories>
    <pluginRepositories>
        <pluginRepository>
            <id>spring-releases</id>
            <url>https://repo.spring.io/libs-release</url>
        </pluginRepository>
    </pluginRepositories>
</project>

With some help from the Internet, we have some simple JavaScript to read the Spring Boot /metrics REST API and fill some values:

HTML and JavaScript (see src/main/resources/static/index.html)

 <h1>Metrics</h1>
<div id="output" name="output"  class="white-frame">
  <ul id="metrics"></ul>  
</div>
<script language="javascript">var myList = document.querySelector('ul');var myRequest = new Request('./metrics/');fetch(myRequest).then(function(response) { return response.json(); }).then(function(data) {for (var i = 0; i < data.length; i++) {var listItem = document.createElement('li');listItem.innerHTML = '<strong>Timestamp' + data[i].timestamp + '</strong>Flow Files Received: ' +data[i].flowfilesreceivedlast5minutes + ' JVM Heap Usage:' + data[i].jvmheap_usage +' Threads Waiting:' + data[i].jvmthread_statestimed_waiting +' Thread Count: ' + data[i].jvmthread_count +' Total Task Duration: ' + data[i].totaltaskdurationnanoseconds +' Bytes Read Last 5 min: ' + data[i].bytesreadlast5minutes +' Flow Files Queued: ' + data[i].flowfilesqueued +' Bytes Queued: ' + data[i].bytesqueued;myList.appendChild(listItem);}});</script>

Example API Calls to Spring Boot

• http://localhost:8090/status/Update
• http://localhost:8090/bulletin/error
• http://localhost:8090/metrics/

REST API for NiFi of Interest

• /nifi-api/flow/process-groups/root/status
• /nifi-api/resources
• /flow/cluster/summary
• /nifi-api/flow/process-groups/root
• /nifi-api/Site-to-site
• /nifi-api/flow/bulletin-board
• /flow/history\?offset\=1\&count\=100
• /nifi-api/flow/search-results\?\q\=NiFi+Operations
• /nifi-api/flow/status
• /flow/process-groups/root/controller-services
• /nifi-api/flow/process-groups/root/status
• /nifi-api/system-diagnostics