DZone
Integration Zone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
  • Refcardz
  • Trend Reports
  • Webinars
  • Zones
  • |
    • Agile
    • AI
    • Big Data
    • Cloud
    • Database
    • DevOps
    • Integration
    • IoT
    • Java
    • Microservices
    • Open Source
    • Performance
    • Security
    • Web Dev
DZone > Integration Zone > API Response Tracking With StreamSets, Elasticsearch, and Kibana

API Response Tracking With StreamSets, Elasticsearch, and Kibana

Learn how to track JSON response data from a RESTful API using Elasticsearch and Kibana to capture and visualize the alerts.

Rathnadevi Manivannan user avatar by
Rathnadevi Manivannan
·
Oct. 15, 17 · Integration Zone · Tutorial
Like (3)
Save
Tweet
11.32K Views

Join the DZone community and get the full member experience.

Join For Free

Overview

RESTful API JSON response data can be used to view various aspects, such as pipeline configuration or monitoring information of the StreamSets Data Collector. This API response information can be used with Data Collector REST API and can be used to provide Data Collector details to a REST-based monitoring system.

In this blog, let us discuss capturing all alerts produced by StreamSets pipelines using RESTful API, loading alerts in Elasticsearch, and visualizing alerts in Kibana.

Pre-requisites

  • Install Java 1.8
  • Install streamsets-datacollector-2.6.0.1

Use Case

Create a data flow pipeline to capture the response of a RESTful API using StreamSets and load it in Elasticsearch.

Synopsis

  • View RESTful API response data
  • Capture RESTful API response
  • Load API response in Elasticsearch
  • Visualize pipeline alerts in Kibana

Viewing RESTful API Response Data

To view RESTful API response data, perform the following:

  • Log in to StreamSets.
  • On the top right corner, click Help icon.
  • Click RESTful API. Different categories such as ACL, definitions, manager, preview, store, and system can be viewed.

select

  • Click manager to view API required to get alerts triggered for all the pipelines.

  • Click try it out! to get the request URL.

select

  • Check the response in UI using the below URL: http://<sdc_host>:/rest/v1/pipelines/alerts

select

Capturing RESTful API Response

To capture RESTful API response, perform the following:

  • Configure HTTP Client Processor by setting Resource URL as “http://<sdc_host>:/rest/v1/pipelines/alerts”, Mode as “Polling”, and Polling Interval.

select

  • Capture RESTful API response using the HTTP client processor.
  • In Pagination tab, set Pagination Mode as “Link HTTP header” and Result Field Path as “/”.

select

Loading API Response in Elasticsearch

To load API Response in Elasticsearch, perform the following:

  • Configure “Elasticsearch” processor.
  • Set Cluster HTTP URI.
  • Use the below template for Elasticsearch:
{
 "template" : "streamsets*",
 "mappings": {
 "uri": {
 "properties": {
  "gauge": {
             "properties": { 
                      "value":{ 
                               "properties": { 
                                        "timestamp": { 
                                        "type":"date", 
                                                 "format":"yyyy-MM-dd HH:mm:ss.SSS||yyyy-MM-
dd'T'HH:mm:ss.SSS'Z'||yyyy-MM-dd||yyyy-MM-dd HH:mm:ss||mmm dd, yyyy HH:mm:ss 
a||epoch_millis" 
        } 
       } 
      } 
     } 
    } 
   } 
  } 
 } 
}

select

Visualizing Pipeline Alerts in Kibana

The alerts produced by all the pipelines can be viewed in Kibana without using StreamSets.

Number of Alerts vs Label as Attributeselectselect

Number of Alerts vs Timestampselect

Conclusion

StreamSets provides different RESTful APIs to get metrics, status, alerts, and so on. These APIs can be used with different visualization tools to visualize data and to monitor the pipelines externally.

References

  • Data Quality Checks with StreamSets using Drift Rules
  • StreamSets Data Collector – Administration
  • StreamSets Data Collector – Elasticsearch
API Elasticsearch Kibana REST Web Protocols Data (computing) Pipeline (software)

Published at DZone with permission of Rathnadevi Manivannan. See the original article here.

Opinions expressed by DZone contributors are their own.

Popular on DZone

  • Top Soft Skills to Identify a Great Software Engineer
  • Ultra-Fast Microservices: When Microstream Meets Wildfly
  • Synchronization Methods for Many-To-Many Associations
  • OpenTelemetry in Action: Identifying Database Dependencies

Comments

Integration Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • MVB Program
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com
  • +1 (919) 678-0300

Let's be friends:

DZone.com is powered by 

AnswerHub logo