Over a million developers have joined DZone.

Hadoop Study Reveals Usage Stats, Benefits, and Challenges

DZone's Guide to

Hadoop Study Reveals Usage Stats, Benefits, and Challenges

· Big Data Zone
Free Resource

Need to build an application around your data? Learn more about dataflow programming for rapid development and greater creativity. 

A new survey on Hadoop suggests that companies using the Apache project's utilities (which include Hadoop Commons, ZooKeeper, HDFS, Hive, MapReduce, etc.) are finding more uses for the open source software and bringing experimental Hadoop projects into production.  102 Hadoop users were surveyed in August by LaunchPad, who was commissioned by Karmasphere.  The study found that nearly 68% of Hadoop projects started in the experimental phase and within a year, 86% moved to active development or production.

The survey also suggests that organizations find the software more useful the longer they use it.  65% of the organizations who used Hadoop for more than a year wrote down more than three reasons for using it.  New users of Hadoop obviously had less knowledge of its usefulness.

These were the top three reasons mentioned for using Hadoop

  1. Mining data for improved Business Intelligence
  2. Reduces the cost of data analysis
  3. Log Analysis

Here were some of the main challenges respondents listed for using Hadoop:

  • Steep learning curve
  • Hiring qualified people
  • Low availability of good products and tooling
  • Not enough information on how to get started

The programming-related challenges included:

  1. Debugging Hadoop jobs
  2. Monitoring Hadoop jobs
  3. Insufficient information about Hadoop
  4. Availability of useful algorithms
  5. Writing Hadoop Jobs

Other areas of the survey found that Hadoop is usually introduced from the developers in an organization rather than management.  

Based  on certain survey questions, LaunchPad projects a 50-60% growth in Hadoop developers for organizations already using Hadoop.  They also expect Java to remain the primary language for Hadoop.  Usage of streaming and the Pig sub-project should remain constant.  Usage of Hive/SQL and Mahout are expected to jump.  Since all of the respondents were current Hadoop users, we can't be sure how many organizations try Hadoop and give up.

Check out the Exaptive data application Studio. Technology agnostic. No glue code. Use what you know and rely on the community for what you don't. Try the community version.


Opinions expressed by DZone contributors are their own.


Dev Resources & Solutions Straight to Your Inbox

Thanks for subscribing!

Awesome! Check your inbox to verify your email so you can start receiving the latest in tech news and resources.


{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}