Customized Alerts for Hadoop Jobs Using Yarn REST API
Apache Ambari takes the guesswork out of operating Hadoop. Using custom Python code and Yarn rest API to find the specific job causing a resource crunch.
Join the DZone community and get the full member experience.
Join For FreeAs per Cloudera, Ambari is a completely open-source management platform for provisioning, managing, monitoring, and securing Apache Hadoop clusters. Apache Ambari takes the guesswork out of operating Hadoop. But one issue which every user encounter while working on Ambari is it doesn’t provide job level alerts. So, while working in a Prod Hadoop cluster some jobs may run more than the expected time or request for resources more than the threshold. If there are numerous jobs, then it becomes challenging to find a specific job that is causing the resource crunch. Users may have to go through each job to identify the root cause.
This issue can be addressed with the use of custom Python code and call to the rest API’s provided by Yarn. Below is the sample simple code that makes use of yarn-rest API to get the job details. You can further customize your code to get the resources allocated and other details.
Prerequisites
- SendGrid Account/Gmail account
Note: If you are using the Gmail SMTP server you need to create an App password which would be enabled after 2-Step Verification as below. The same password can be used in server.login
- JSON (to parse the JSON response from yarn rest-API), urlib package for opening and reading URLs
- email and SMTP libraries to send emails.
Below is the python code which can be used to get the alerts for the Hadoop jobs. You can further customize the code to get other details (Just you need to parse the JSON response). You can refer this link
import json, urllib.request
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText
import smtplib
#Email function
def send_email(test):
loginuser = "send grid user"
loginpassword="send grid password"
fromaddr="source email "
toaddr = "target email id"
msg = MIMEMultipart()
msg['From'] = fromaddr
msg['To'] = toaddr
msg['Subject'] = "Alert"
body = "The list of long running job(s) are " + test
msg.attach(MIMEText(body, 'plain'))
server = smtplib.SMTP('smtp.sendgrid.net', 587)
server.starttls()
server.login(loginuser,loginpassword)
text = msg.as_string()
server.sendmail(fromaddr, toaddr, text)
server.quit()
link="http://<rm host name>:8088/ws/v1/cluster/apps?states=RUNNING"
#Mention your threshold time in milliseconds
Prescribed_limit=6000
with urllib.request.urlopen(link) as response:
result=json.loads(response.read().decode('utf8'))
# Below to parse the json
for jobs in result['apps']['app']:
if jobs['elapsedTime']>prescribed_limit:
send_email(str("\nApp Name: {}".format(jobs['name']) +" with Application id: {}".format(jobs['id'])+" running for {} hours".format(round(jobs['elapsedTime']/1000/60/60))))
Opinions expressed by DZone contributors are their own.
Trending
-
Using Render Log Streams to Log to Papertrail
-
Tech Hiring: Trends, Predictions, and Strategies for Success
-
Alpha Testing Tutorial: A Comprehensive Guide With Best Practices
-
Knowing and Valuing Apache Kafka’s ISR (In-Sync Replicas)
Comments