DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports Events Over 2 million developers have joined DZone. Join Today! Thanks for visiting DZone today,
Edit Profile Manage Email Subscriptions Moderation Admin Console How to Post to DZone Article Submission Guidelines
View Profile
Sign Out
Refcards
Trend Reports
Events
Zones
Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Partner Zones AWS Cloud
by AWS Developer Relations
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Partner Zones
AWS Cloud
by AWS Developer Relations

Trending

  • Using Render Log Streams to Log to Papertrail
  • Tech Hiring: Trends, Predictions, and Strategies for Success
  • Alpha Testing Tutorial: A Comprehensive Guide With Best Practices
  • Knowing and Valuing Apache Kafka’s ISR (In-Sync Replicas)

Trending

  • Using Render Log Streams to Log to Papertrail
  • Tech Hiring: Trends, Predictions, and Strategies for Success
  • Alpha Testing Tutorial: A Comprehensive Guide With Best Practices
  • Knowing and Valuing Apache Kafka’s ISR (In-Sync Replicas)
  1. DZone
  2. Software Design and Architecture
  3. Integration
  4. Customized Alerts for Hadoop Jobs Using Yarn REST API

Customized Alerts for Hadoop Jobs Using Yarn REST API

Apache Ambari takes the guesswork out of operating Hadoop. Using custom Python code and Yarn rest API to find the specific job causing a resource crunch.

Subash Konar user avatar by
Subash Konar
·
Updated Jul. 20, 20 · Code Snippet
Like (3)
Save
Tweet
Share
4.96K Views

Join the DZone community and get the full member experience.

Join For Free

As per Cloudera, Ambari is a completely open-source management platform for provisioning, managing, monitoring, and securing Apache Hadoop clusters. Apache Ambari takes the guesswork out of operating Hadoop. But one issue which every user encounter while working on Ambari is it doesn’t provide job level alerts. So, while working in a Prod Hadoop cluster some jobs may run more than the expected time or request for resources more than the threshold. If there are numerous jobs, then it becomes challenging to find a specific job that is causing the resource crunch. Users may have to go through each job to identify the root cause.

This issue can be addressed with the use of custom Python code and call to the rest API’s provided by Yarn. Below is the sample simple code that makes use of yarn-rest API to get the job details. You can further customize your code to get the resources allocated and other details.

Prerequisites

  • SendGrid Account/Gmail account

Note: If you are using the Gmail SMTP server you need to create an App password which would be enabled after 2-Step Verification as below. The same password can be used in server.login

log in with 2 step verification

  • JSON (to parse the JSON response from yarn rest-API), urlib package for opening and reading URLs
  • email and SMTP libraries to send emails.

Below is the python code which can be used to get the alerts for the Hadoop jobs. You can further customize the code to get other details (Just you need to parse the JSON response). You can refer this link

https://hadoop.apache.org/docs/r2.7.3/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_Applications_APISubmit_Application


Python
x
34
 
1
import json, urllib.request
2
from email.mime.multipart import MIMEMultipart
3
from email.mime.text import MIMEText
4
import smtplib
5
6
7
#Email function
8
def send_email(test):
9
 loginuser = "send grid user"
10
 loginpassword="send grid password"
11
 fromaddr="source email "
12
 toaddr = "target email id"
13
 msg = MIMEMultipart()
14
 msg['From'] = fromaddr
15
 msg['To'] = toaddr
16
 msg['Subject'] = "Alert"
17
 body = "The list of long running job(s) are " + test
18
 msg.attach(MIMEText(body, 'plain'))
19
 server = smtplib.SMTP('smtp.sendgrid.net', 587)
20
 server.starttls()
21
 server.login(loginuser,loginpassword)
22
 text = msg.as_string()
23
 server.sendmail(fromaddr, toaddr, text)
24
 server.quit()
25
26
link="http://<rm host name>:8088/ws/v1/cluster/apps?states=RUNNING"
27
#Mention your threshold time in milliseconds
28
Prescribed_limit=6000
29
with urllib.request.urlopen(link) as response:
30
 result=json.loads(response.read().decode('utf8'))
31
# Below to parse the json
32
for jobs in result['apps']['app']:
33
    if jobs['elapsedTime']>prescribed_limit:
34
       send_email(str("\nApp Name: {}".format(jobs['name']) +" with Application id: {}".format(jobs['id'])+" running for {} hours".format(round(jobs['elapsedTime']/1000/60/60))))


hadoop API career REST Web Protocols

Opinions expressed by DZone contributors are their own.

Trending

  • Using Render Log Streams to Log to Papertrail
  • Tech Hiring: Trends, Predictions, and Strategies for Success
  • Alpha Testing Tutorial: A Comprehensive Guide With Best Practices
  • Knowing and Valuing Apache Kafka’s ISR (In-Sync Replicas)

Comments

Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com

Let's be friends: