DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports
Events Video Library
Over 2 million developers have joined DZone. Join Today! Thanks for visiting DZone today,
Edit Profile Manage Email Subscriptions Moderation Admin Console How to Post to DZone Article Submission Guidelines
View Profile
Sign Out
Refcards
Trend Reports
Events
View Events Video Library
Zones
Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks

Integrating PostgreSQL Databases with ANF: Join this workshop to learn how to create a PostgreSQL server using Instaclustr’s managed service

Mobile Database Essentials: Assess data needs, storage requirements, and more when leveraging databases for cloud and edge applications.

Monitoring and Observability for LLMs: Datadog and Google Cloud discuss how to achieve optimal AI model performance.

Automated Testing: The latest on architecture, TDD, and the benefits of AI and low-code tools.

Related

  • MapReduce and Yarn Part 2: Hadoop Processing Unit
  • MapReduce and Yarn: Hadoop Processing Unit Part 1
  • Apache Hadoop: How MapReduce Can Essentiate Data From HDFS
  • How to Execute Distributed MapReduce on Java Over Data Stored in Redis

Trending

  • Exploring Sorting Algorithms: A Comprehensive Guide
  • JWT Token Revocation: Centralized Control vs. Distributed Kafka Handling
  • Automated Testing Lifecycle
  • Time Series Analysis: VAR-Model-As-A-Service Using Flask and MinIO
  1. DZone
  2. Data Engineering
  3. Big Data
  4. The Evolution of MapReduce and Hadoop

The Evolution of MapReduce and Hadoop

Adam Diaz user avatar by
Adam Diaz
·
Oct. 02, 14 · Interview
Like (0)
Save
Tweet
Share
6.21K Views

Join the DZone community and get the full member experience.

Join For Free

cover Recently I authored a section of the DZone Guide for Big Data 2014. I wrote about MapReduce and the evolution of Hadoop. There is a ton of buzz in the market about speed to insight and the push toward alternative DAG engines like Spark. I see these new techs as exciting and awesome. I welcome innovation and creativity in the Hadoop market. I also temper this excitement with a bit of reality. The reality is that technology has a maturity curve. It comes out from the huge brain incubators like Google to the rest of us. Much like MapReduce, HDFS and Hbase these technologies have a long road until they grow into a vibrant open source community with dedicated developers. Then comes a stage where the project may become its own incubator project within Apache or other open source framework. Finally, you may see the tech evolve to the point that its included in a supported Hadoop distribution. Now in their efforts to compete you will see vendors clamor over one another to be the first to include hot tech brand X in their offering. I see this as competition and healthy but I will say this could lead to the inclusion of technology that we may not see removed from a distro immediately but it may just fade away. Much like RedHat many years ago with a Linux distribution, Hadoop distributions are still relatively new. Their choices are still small. Eventually, hopefully, we will see the number of projects grow to the point where vendors are actually refusing or removing projects that are not longer relevant. I’m not sure we are there yet. All that said do I think MapReduce is going to dry up and blow away? No I don’t. Do I think Spark and alternative engines will gain steam as they prove their efficiencies? Yes I do. I think most folks need to MRandBigDatahear more than one group validate scalability. I’m sure Google has tooling that works for them. Are they sharing it or better yet has it matured to the point that we mere mortals can use it? I think that remains to be seen. MapReduce and further more Hive which uses MapReduce has a HUGE ecosystem of tools that depend on Hive.Also take a look at Stinger Next to see some exciting new developments of Hive. Take a walk down the rows of tables of vendors at the next Hadoop Summit or Strata and ask them in detail which Hadoop tool their tool depends. Most likely you will eventually get to an answer that basically is Hive (or Hcatalog). Hive and MapReduce will be here for years to come if for no other reason than market penetration. Anyway, take a look at the new DZone Guide. Its has many good topics that may interest you.

hadoop MapReduce

Published at DZone with permission of , DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

Related

  • MapReduce and Yarn Part 2: Hadoop Processing Unit
  • MapReduce and Yarn: Hadoop Processing Unit Part 1
  • Apache Hadoop: How MapReduce Can Essentiate Data From HDFS
  • How to Execute Distributed MapReduce on Java Over Data Stored in Redis

Comments

Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 100
  • Nashville, TN 37211
  • support@dzone.com

Let's be friends: