DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports Events Over 2 million developers have joined DZone. Join Today! Thanks for visiting DZone today,
Edit Profile Manage Email Subscriptions Moderation Admin Console How to Post to DZone Article Submission Guidelines
View Profile
Sign Out
Refcards
Trend Reports
Events
Zones
Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
  1. DZone
  2. Coding
  3. Frameworks
  4. Simple Apache NiFi Operations Dashboard (Part 2): Spring Boot

Simple Apache NiFi Operations Dashboard (Part 2): Spring Boot

In this post, we continue with uilding a dashboard with the open source big data platform Apache NiFi, using Spring Boot 2.0.6.

Tim Spann user avatar by
Tim Spann
CORE ·
Oct. 24, 18 · Tutorial
Like (9)
Save
Tweet
Share
12.69K Views

Join the DZone community and get the full member experience.

Join For Free

If you missed Part 1, you can check it out here.

Simple Apache NiFi Operations Dashboard - Part 2

To access data to display in our dashboard we will use some Spring Boot 2.06 Java 8 microservices to call Apache Hive 3.1.0 tables in HDP 3.0 on Hadoop 3.1.

We will have our website hosted and make REST Calls to Apache NiFi, our microservices, YARN, and other APIs.

As you can see we can easily incorporate data from HDP 3 — Apache Hive 3.1.0 in Spring Boot Java applications with not much trouble. You can see the Maven build script (all code is in GitHub).

Our motivation is to put all this data somewhere and show it on a dashboard that can use REST APIs for data access and updates. We may choose to use Apache NiFi for all REST APIs or we can do some in Apache NiFi. We are still exploring. We can also decide to change the backend to HBase 2.0, Phoenix, Druid or a combination of these. We will see.

Spring Boot 2.0.6 Loading

JSON Output


Spring Boot Microservices and UI

https://github.com/tspannhw/operations-dashboard

To start, I have a simple web page that calls one of the REST APIs.

The microservice can be run off of YARN 3.1, Kubernetes, CloudFoundry, OpenShift, or any machine that can run a simple Java 8 jar.

We can have this HTML as part of a larger dashboard or hosted anywhere.

For parsing the monitoring data we have some schemas for metrics, status, and bulletins.

Now that I'm monitoring data is in Apache Hive, I can query it with ease in Apache Zeppelin (or any JDBC/ODBC tool).

Apache Zeppelin Screens

We have a lot of reporting tasks for Monitoring NiFi.

We read from NiFi and send to NiFi, would be nice to have a dedicated reporting cluster.

Just Show Me Bulletins for MonitorMemory (You Can See That in Reporting Tasks)



NiFi Query To Limit Which Bulletins We Are Storing In Hive (For Now Just grab Errors)

Spring Boot Code for REST APIs

Metrics REST API Results

Bulletin REST API Results

Metrics Home Page

Run the Microservice

java -Xms512m -Xmx2048m -Dhdp.version=3.0.0 -Djava.net.preferIPv4Stack=true -jar target/operations-0.0.1-SNAPSHOT.jar

Maven POM

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.dataflowdeveloper</groupId>
<artifactId>operations</artifactId>
<version>0.0.1-SNAPSHOT</version>
<packaging>jar</packaging>
<name>operations</name>
<description>Apache Hive Operations Spring Boot</description>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.0.5.RELEASE</version>
<relativePath/>
</parent>


<properties>
<java.version>1.8</java.version>
</properties>


<dependencies>


<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-web</artifactId>
    <exclusions>
        <exclusion>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-logging</artifactId>
        </exclusion>
<exclusion>
 <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-tomcat</artifactId>
 </exclusion>
    </exclusions>
</dependency>
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-log4j2</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-jdbc</artifactId>
</dependency>
<dependency>
<groupId>org.apache.hive</groupId>
<artifactId>hive-jdbc</artifactId>
<version>3.1.0</version>
<exclusions>
<exclusion>
<groupId>org.eclipse.jetty.aggregate</groupId>
<artifactId>*</artifactId>
</exclusion>
<exclusion>
<artifactId>servlet-api</artifactId>
<groupId>javax.servlet</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework.restdocs</groupId>
<artifactId>spring-restdocs-mockmvc</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
</plugins>
</build>
   <repositories>
        <repository>
            <id>spring-releases</id>
            <url>https://repo.spring.io/libs-release</url>
        </repository>
    </repositories>
    <pluginRepositories>
        <pluginRepository>
            <id>spring-releases</id>
            <url>https://repo.spring.io/libs-release</url>
        </pluginRepository>
    </pluginRepositories>
</project>

With some help from the Internet, we have some simple JavaScript to read the Spring Boot /metrics REST API and fill some values:

HTML and JavaScript (see src/main/resources/static/index.html)

 <h1>Metrics</h1>
<div id="output" name="output"  class="white-frame">
  <ul id="metrics"></ul>  
</div>
<script language="javascript">var myList = document.querySelector('ul');var myRequest = new Request('./metrics/');fetch(myRequest).then(function(response) { return response.json(); }).then(function(data) {for (var i = 0; i < data.length; i++) {var listItem = document.createElement('li');listItem.innerHTML = '<strong>Timestamp' + data[i].timestamp + '</strong>Flow Files Received: ' +data[i].flowfilesreceivedlast5minutes + ' JVM Heap Usage:' + data[i].jvmheap_usage +' Threads Waiting:' + data[i].jvmthread_statestimed_waiting +' Thread Count: ' + data[i].jvmthread_count +' Total Task Duration: ' + data[i].totaltaskdurationnanoseconds +' Bytes Read Last 5 min: ' + data[i].bytesreadlast5minutes +' Flow Files Queued: ' + data[i].flowfilesqueued +' Bytes Queued: ' + data[i].bytesqueued;myList.appendChild(listItem);}});</script>


Example API Calls to Spring Boot

  • http://localhost:8090/status/Update
  • http://localhost:8090/bulletin/error
  • http://localhost:8090/metrics/

REST API for NiFi of Interest

  • /nifi-api/flow/process-groups/root/status
  • /nifi-api/resources
  • /flow/cluster/summary
  • /nifi-api/flow/process-groups/root
  • /nifi-api/Site-to-site
  • /nifi-api/flow/bulletin-board
  • /flow/history\?offset\=1\&count\=100
  • /nifi-api/flow/search-results\?\q\=NiFi+Operations
  • /nifi-api/flow/status
  • /flow/process-groups/root/controller-services
  • /nifi-api/flow/process-groups/root/status
  • /nifi-api/system-diagnostics
Spring Framework Spring Boot Apache NiFi Dashboard (Mac OS)

Opinions expressed by DZone contributors are their own.

Popular on DZone

  • Comparing Kubernetes Gateway and Ingress APIs
  • How To Generate Code Coverage Report Using JaCoCo-Maven Plugin
  • Tech Layoffs [Comic]
  • Why the World Is Moving Towards Serverless Computing

Comments

Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com
  • +1 (919) 678-0300

Let's be friends: