DZone
Big Data Zone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
  • Refcardz
  • Trend Reports
  • Webinars
  • Zones
  • |
    • Agile
    • AI
    • Big Data
    • Cloud
    • Database
    • DevOps
    • Integration
    • IoT
    • Java
    • Microservices
    • Open Source
    • Performance
    • Security
    • Web Dev
DZone > Big Data Zone > Simple Apache NiFi Operations Dashboard (Part 2): Spring Boot

Simple Apache NiFi Operations Dashboard (Part 2): Spring Boot

In this post, we continue with uilding a dashboard with the open source big data platform Apache NiFi, using Spring Boot 2.0.6.

Tim Spann user avatar by
Tim Spann
CORE ·
Oct. 24, 18 · Big Data Zone · Tutorial
Like (9)
Save
Tweet
12.25K Views

Join the DZone community and get the full member experience.

Join For Free

If you missed Part 1, you can check it out here.

Simple Apache NiFi Operations Dashboard - Part 2

To access data to display in our dashboard we will use some Spring Boot 2.06 Java 8 microservices to call Apache Hive 3.1.0 tables in HDP 3.0 on Hadoop 3.1.

We will have our website hosted and make REST Calls to Apache NiFi, our microservices, YARN, and other APIs.

As you can see we can easily incorporate data from HDP 3 — Apache Hive 3.1.0 in Spring Boot Java applications with not much trouble. You can see the Maven build script (all code is in GitHub).

Our motivation is to put all this data somewhere and show it on a dashboard that can use REST APIs for data access and updates. We may choose to use Apache NiFi for all REST APIs or we can do some in Apache NiFi. We are still exploring. We can also decide to change the backend to HBase 2.0, Phoenix, Druid or a combination of these. We will see.

Spring Boot 2.0.6 Loading

JSON Output


Spring Boot Microservices and UI

https://github.com/tspannhw/operations-dashboard

To start, I have a simple web page that calls one of the REST APIs.

The microservice can be run off of YARN 3.1, Kubernetes, CloudFoundry, OpenShift, or any machine that can run a simple Java 8 jar.

We can have this HTML as part of a larger dashboard or hosted anywhere.

For parsing the monitoring data we have some schemas for metrics, status, and bulletins.

Now that I'm monitoring data is in Apache Hive, I can query it with ease in Apache Zeppelin (or any JDBC/ODBC tool).

Apache Zeppelin Screens

We have a lot of reporting tasks for Monitoring NiFi.

We read from NiFi and send to NiFi, would be nice to have a dedicated reporting cluster.

Just Show Me Bulletins for MonitorMemory (You Can See That in Reporting Tasks)



NiFi Query To Limit Which Bulletins We Are Storing In Hive (For Now Just grab Errors)

Spring Boot Code for REST APIs

Metrics REST API Results

Bulletin REST API Results

Metrics Home Page

Run the Microservice

java -Xms512m -Xmx2048m -Dhdp.version=3.0.0 -Djava.net.preferIPv4Stack=true -jar target/operations-0.0.1-SNAPSHOT.jar

Maven POM

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.dataflowdeveloper</groupId>
<artifactId>operations</artifactId>
<version>0.0.1-SNAPSHOT</version>
<packaging>jar</packaging>
<name>operations</name>
<description>Apache Hive Operations Spring Boot</description>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.0.5.RELEASE</version>
<relativePath/>
</parent>


<properties>
<java.version>1.8</java.version>
</properties>


<dependencies>


<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-web</artifactId>
    <exclusions>
        <exclusion>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-logging</artifactId>
        </exclusion>
<exclusion>
 <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-tomcat</artifactId>
 </exclusion>
    </exclusions>
</dependency>
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-log4j2</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-jdbc</artifactId>
</dependency>
<dependency>
<groupId>org.apache.hive</groupId>
<artifactId>hive-jdbc</artifactId>
<version>3.1.0</version>
<exclusions>
<exclusion>
<groupId>org.eclipse.jetty.aggregate</groupId>
<artifactId>*</artifactId>
</exclusion>
<exclusion>
<artifactId>servlet-api</artifactId>
<groupId>javax.servlet</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework.restdocs</groupId>
<artifactId>spring-restdocs-mockmvc</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
</plugins>
</build>
   <repositories>
        <repository>
            <id>spring-releases</id>
            <url>https://repo.spring.io/libs-release</url>
        </repository>
    </repositories>
    <pluginRepositories>
        <pluginRepository>
            <id>spring-releases</id>
            <url>https://repo.spring.io/libs-release</url>
        </pluginRepository>
    </pluginRepositories>
</project>

With some help from the Internet, we have some simple JavaScript to read the Spring Boot /metrics REST API and fill some values:

HTML and JavaScript (see src/main/resources/static/index.html)

 <h1>Metrics</h1>
<div id="output" name="output"  class="white-frame">
  <ul id="metrics"></ul>  
</div>
<script language="javascript">var myList = document.querySelector('ul');var myRequest = new Request('./metrics/');fetch(myRequest).then(function(response) { return response.json(); }).then(function(data) {for (var i = 0; i < data.length; i++) {var listItem = document.createElement('li');listItem.innerHTML = '<strong>Timestamp' + data[i].timestamp + '</strong>Flow Files Received: ' +data[i].flowfilesreceivedlast5minutes + ' JVM Heap Usage:' + data[i].jvmheap_usage +' Threads Waiting:' + data[i].jvmthread_statestimed_waiting +' Thread Count: ' + data[i].jvmthread_count +' Total Task Duration: ' + data[i].totaltaskdurationnanoseconds +' Bytes Read Last 5 min: ' + data[i].bytesreadlast5minutes +' Flow Files Queued: ' + data[i].flowfilesqueued +' Bytes Queued: ' + data[i].bytesqueued;myList.appendChild(listItem);}});</script>


Example API Calls to Spring Boot

  • http://localhost:8090/status/Update
  • http://localhost:8090/bulletin/error
  • http://localhost:8090/metrics/

REST API for NiFi of Interest

  • /nifi-api/flow/process-groups/root/status
  • /nifi-api/resources
  • /flow/cluster/summary
  • /nifi-api/flow/process-groups/root
  • /nifi-api/Site-to-site
  • /nifi-api/flow/bulletin-board
  • /flow/history\?offset\=1\&count\=100
  • /nifi-api/flow/search-results\?\q\=NiFi+Operations
  • /nifi-api/flow/status
  • /flow/process-groups/root/controller-services
  • /nifi-api/flow/process-groups/root/status
  • /nifi-api/system-diagnostics
Spring Framework Spring Boot Apache NiFi Dashboard (Mac OS)

Opinions expressed by DZone contributors are their own.

Popular on DZone

  • Replace your Scripts with Gradle Tasks
  • Resilient Kafka Consumers With Reactor Kafka
  • How Do You Know If a Graph Database Solves the Problem?
  • Build a Java Microservice With AuraDB Free

Comments

Big Data Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • MVB Program
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com
  • +1 (919) 678-0300

Let's be friends:

DZone.com is powered by 

AnswerHub logo