{{announcement.body}}
{{announcement.title}}

Customized Alerts for Hadoop Jobs Using Yarn REST API

DZone 's Guide to

Customized Alerts for Hadoop Jobs Using Yarn REST API

Apache Ambari takes the guesswork out of operating Hadoop. Using custom Python code and Yarn rest API to find the specific job causing a resource crunch.

· Big Data Zone ·
Free Resource

As per Cloudera, Ambari is a completely open-source management platform for provisioning, managing, monitoring, and securing Apache Hadoop clusters. Apache Ambari takes the guesswork out of operating Hadoop. But one issue which every user encounter while working on Ambari is it doesn’t provide job level alerts. So, while working in a Prod Hadoop cluster some jobs may run more than the expected time or request for resources more than the threshold. If there are numerous jobs, then it becomes challenging to find a specific job that is causing the resource crunch. Users may have to go through each job to identify the root cause.

This issue can be addressed with the use of custom Python code and call to the rest API’s provided by Yarn. Below is the sample simple code that makes use of yarn-rest API to get the job details. You can further customize your code to get the resources allocated and other details.

Prerequisites

  • SendGrid Account/Gmail account

Note: If you are using the Gmail SMTP server you need to create an App password which would be enabled after 2-Step Verification as below. The same password can be used in server.login

log in with 2 step verification

  • JSON (to parse the JSON response from yarn rest-API), urlib package for opening and reading URLs
  • email and SMTP libraries to send emails.

Below is the python code which can be used to get the alerts for the Hadoop jobs. You can further customize the code to get other details (Just you need to parse the JSON response). You can refer this link

https://hadoop.apache.org/docs/r2.7.3/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_Applications_APISubmit_Application


Python


Topics:
big data, email, hadoop, python, yarn

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}