Over a million developers have joined DZone.

[YARN] : NullPointerException in MRClientService.getHttpPort()

· Cloud Zone

Download the Essential Cloud Buyer’s Guide to learn important factors to consider before selecting a provider as well as buying criteria to help you make the best decision for your infrastructure needs, brought to you in partnership with Internap.

Have you moved your Hadoop jobs over to Yarn?  Are you seeing the following NullPointerException coming out of your job?
Caused by: java.lang.NullPointerException
 at org.apache.hadoop.mapreduce.v2.app.client.MRClientService.getHttpPort(MRClientService.java:167)
 at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:150)
 ... 14 more
This appears to be a poorly handled case in Hadoop, where the webApp fails to load.  You need to look at the full log file.  You will likely see the following in the log preceeding the NPE:
ERROR (org.apache.hadoop.mapreduce.v2.app.client.MRClientService:139) - Webapps failed to start. Ignoring for now:
That line is produced by the following code in YARN:
   LOG.info("Instantiated MRClientService at " + this.bindAddress);
    try {
      webApp = WebApps.$for("mapreduce", AppContext.class, appContext, "ws").with(conf).
          start(new AMWebApp());
    } catch (Exception e) {
      LOG.error("Webapps failed to start. Ignoring for now:", e);
This means your webApp failed to load.  In my case, this was caused by dependency conflicts on my classpath.  (servlet & jasper compiler)  I've also seen it cause by a dependency mssing, such as:
java.lang.ClassNotFoundException: Class org.apache.hadoop.yarn.server.webproxy.amfilter.AmFilterInitializer
In my situation, the solution was to cleanup my classpath, but the webApp might fail for lots of reasons, which will result in an NPE later in the log file when MRClientService attempts to getHttpPort without a webApp, which is shown in the following code:
public int getHttpPort() {
    return webApp.port();
Anyway, that is probably more detail than you wanted. Happy yarning.

The Cloud Zone is brought to you in partnership with Internap. Read Bare-Metal Cloud 101 to learn about bare-metal cloud and how it has emerged as a way to complement virtualized services.


Published at DZone with permission of Brian O' Neill, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

The best of DZone straight to your inbox.

Please provide a valid email address.

Thanks for subscribing!

Awesome! Check your inbox to verify your email so you can start receiving the latest in tech news and resources.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}