Over a million developers have joined DZone.
Platinum Partner

Configuring and Tuning MapReduce's Shuffle

· Database Zone

Speed up your development and test cycles with fully automated data delivery, view a 10 minute demo, brought to you in partnership with Delphix.

Once you have outgrown your small Hadoop cluster it’s worth tuning some of the shuffle configurables to ensure that your performance keeps up with the physical growth of your cluster. The figure below shows key configurables in the shuffle stage in Hadoop versions 1.x and earlier, and identifies those that should be tuned.

You can read more about these configurables and their default values by looking at mapred-default.xml. My book Hadoop in Practice (Manning Publications) in chapter 6 discusses how some of the configuration values in the figure should be tweaked when you start working with mid to large-size Hadoop clusters.

The Database Zone is brought to you in partnership with Delphix. Learn how you can easily virtualize and clone your database, read the white paper.

Topics:

Published at DZone with permission of Alex Holmes , DZone MVB .

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}