Customized Alerts for Hadoop Jobs Using Yarn REST API

As per Cloudera, Ambari is a completely open-source management platform for provisioning, managing, monitoring, and securing Apache Hadoop clusters. Apache Ambari takes the guesswork out of operating Hadoop. But one issue which every user encounter while working on Ambari is it doesn’t provide job level alerts. So, while working in a Prod Hadoop cluster some jobs may run more than the expected time or request for resources more than the threshold. If there are numerous jobs, then it becomes challenging to find a specific job that is causing the resource crunch. Users may have to go through each job to identify the root cause.

This issue can be addressed with the use of custom Python code and call to the rest API’s provided by Yarn. Below is the sample simple code that makes use of yarn-rest API to get the job details. You can further customize your code to get the resources allocated and other details.

Prerequisites

• SendGrid Account/Gmail account

Note: If you are using the Gmail SMTP server you need to create an App password which would be enabled after 2-Step Verification as below. The same password can be used in server.login

• JSON (to parse the JSON response from yarn rest-API), urlib package for opening and reading URLs
• email and SMTP libraries to send emails.

Below is the python code which can be used to get the alerts for the Hadoop jobs. You can further customize the code to get other details (Just you need to parse the JSON response). You can refer this link

https://hadoop.apache.org/docs/r2.7.3/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_Applications_APISubmit_Application

import json, urllib.request

from email.mime.multipart import MIMEMultipart

from email.mime.text import MIMEText

import smtplib

#Email function

def send_email(test):

 loginuser = "send grid user"

 loginpassword="send grid password"

 fromaddr="source email "

 toaddr = "target email id"

 msg = MIMEMultipart()

 msg['From'] = fromaddr

 msg['To'] = toaddr

 msg['Subject'] = "Alert"

 body = "The list of long running job(s) are " + test

 msg.attach(MIMEText(body, 'plain'))

 server = smtplib.SMTP('smtp.sendgrid.net', 587)

 server.starttls()

 server.login(loginuser,loginpassword)

 text = msg.as_string()

 server.sendmail(fromaddr, toaddr, text)

 server.quit()

link="http://<rm host name>:8088/ws/v1/cluster/apps?states=RUNNING"

#Mention your threshold time in milliseconds

Prescribed_limit=6000

with urllib.request.urlopen(link) as response:

 result=json.loads(response.read().decode('utf8'))

# Below to parse the json

for jobs in result['apps']['app']:

    if jobs['elapsedTime']>prescribed_limit:

       send_email(str("\nApp Name: {}".format(jobs['name']) +" with Application id: {}".format(jobs['id'])+" running for {} hours".format(round(jobs['elapsedTime']/1000/60/60))))