DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports
Events Video Library
Refcards
Trend Reports

Events

View Events Video Library

Zones

Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks

The software you build is only as secure as the code that powers it. Learn how malicious code creeps into your software supply chain.

Apache Cassandra combines the benefits of major NoSQL databases to support data management needs not covered by traditional RDBMS vendors.

Generative AI has transformed nearly every industry. How can you leverage GenAI to improve your productivity and efficiency?

Modernize your data layer. Learn how to design cloud-native database architectures to meet the evolving demands of AI and GenAI workloads.

Related

  • How To Check for JSON Insecure Deserialization (JID) Attacks With Java
  • Why Use LocalPV with NVMe for Your Workload?
  • Page Object Model for Performance Testing With Gatling
  • OWASP TOP 10 API Security Part 2 (Broken Object Level Authorization)

Trending

  • How to Ensure Cross-Time Zone Data Integrity and Consistency in Global Data Pipelines
  • Secure by Design: Modernizing Authentication With Centralized Access and Adaptive Signals
  • Operational Principles, Architecture, Benefits, and Limitations of Artificial Intelligence Large Language Models
  • The Smart Way to Talk to Your Database: Why Hybrid API + NL2SQL Wins
  1. DZone
  2. Culture and Methodologies
  3. Career Development
  4. Using YARN API to Determine Resources Available for Spark Application Submission: Part II

Using YARN API to Determine Resources Available for Spark Application Submission: Part II

Using the YARN REST API for submitting Spark applications and determining available resources. Applies to all Apache Hadoop distributions.

By 
Rachel Warren user avatar
Rachel Warren
·
Nov. 07, 16 · Opinion
Likes (2)
Comment
Save
Tweet
Share
4.8K Views

Join the DZone community and get the full member experience.

Join For Free

Welcome to the second part of this introduction to the YARN API. Today I will walk through the resources available for a job and how to find your queue.

How Many Total Resources Are Available for My Job?

When configuring a spark driver, each driver and executor cannot be larger than the size of one Yarn container. Additionally, the total resources requested in terms of both CPU and memory cannot be larger than all the resources available on the cluster to the user submitting the job. In other words, we can consider the resource requirements to be governed by the following equations.

Total vCores Available ≥ spark.executor.cores ✕ spark.executor.instances + 1

(Where the 1 is for the one core required by the driver)

Total_MB_Available ≥(spark.executor.memory + executor memory overhead)
✕ spark.executor.instances
+ spark.driver.memory + driver memory overhead

Securing this information particularly in a system with queuing configured and other active users can be a little complicated.

The behavior of a job that is submitted with too many total resources seems to differ depending on the user, cluster, and how excessive the request is. My experience has been that if the resources requested are larger than the size of the cluster, according to the configuration, the job will fail with a meaningful error. If however a Spark Application requests more resources than are available on the cluster at the time of submission, or more resources than are available in this queue (but not in the cluster), but less than the size of the whole cluster, the job may hang in an “unassigned” state until such time (in many cases never) those resources become available. Because my program requires configuring Spark jobs submitted by users of our applications, who often do not themselves have access to the resource manager and thus don’t have a way to learn about the traffic on their cluster, it is important to avoid this case. It is possible that in a different scenario making a large request and waiting for resources to be available might be desirable.

Furthermore, here is what I learned about querying the Yarn Scheduler API to determine cluster burn. I am going to try to keep the explanation simple here, but I found this blog really useful for me when I was trying to understand the yarn scheduler.

The Yarn scheduler API has a “scheduler” object. Inside the scheduler object is a “schedulerInfo” object. The schedulerInfo object has a queue “type.” The “type” is the kind of queue, either “capacityScheduler” or “fairScheduler.” The rest of the structure of the object is different depending on the queue type. The values shown on the official documentation page are for the capcaityScheduler.

Find queue type from the scheduler API API Call: http://<rm http address:port>/ws/v1/cluster/scheduler Path to Json Scheduler

→ schedulerInfo

   → type

→ (will be either “capacityScheduler” or “fairScheduler” ) 

How to Find Your Queue

Each cluster has a root queue. Thus, the “schedulerInfo” object has a field called “rootQueue.” ‘rootQueue’ contains a nested object of queues.

Find the root queue from the Scheduler API API Call: http://<rm http address:port>/ws/v1/cluster/scheduler Path to Json scheduler

   → schedulerInfo

      → rootQueue

→ queues

Note: This is another place where my experience differed from the documentation. I could not access the “queues” object directly, I had to get to the “rootQueue” first.

The queues are in tree structure, so you can search through the queues (I used a breadth first search) to find the one whose name corresponds to the name of the user submitting the application. In some instances, a queue with this name will not exist. In this case, the resources allocated for this job will be specified by the “default” queue, so you will need to do a second search to find the default queue object. In the case of the fair scheduler, the user will not have a queue until they have submitted one job. The first time they submit they will get the resources specified by the default queue. In subsequent submissions, the “queues” object will contain a queue corresponding to their username.

In the next blog, I will walk through the difference between the capacity scheduler and the fair scheduler in addition to how to calculate your memory overhead. Stay tuned for more information!

API application cluster job scheduling Object (computer science) career

Published at DZone with permission of Rachel Warren, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

Related

  • How To Check for JSON Insecure Deserialization (JID) Attacks With Java
  • Why Use LocalPV with NVMe for Your Workload?
  • Page Object Model for Performance Testing With Gatling
  • OWASP TOP 10 API Security Part 2 (Broken Object Level Authorization)

Partner Resources

×

Comments

The likes didn't load as expected. Please refresh the page and try again.

ABOUT US

  • About DZone
  • Support and feedback
  • Community research
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Core Program
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 100
  • Nashville, TN 37211
  • support@dzone.com

Let's be friends: