DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports Events Over 2 million developers have joined DZone. Join Today! Thanks for visiting DZone today,
Edit Profile Manage Email Subscriptions Moderation Admin Console How to Post to DZone Article Submission Guidelines
View Profile
Sign Out
Refcards
Trend Reports
Events
Zones
Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
  1. DZone
  2. Data Engineering
  3. Big Data
  4. Introducing Hadoop (HDFS) Connector v5.0.0

Introducing Hadoop (HDFS) Connector v5.0.0

This article serves as a quick demo using Hadoop Connector.

Nathan Nam user avatar by
Nathan Nam
·
Oct. 20, 16 · Tutorial
Like (2)
Save
Tweet
Share
4.75K Views

Join the DZone community and get the full member experience.

Join For Free

According to a recent survey conducted by DNV GL – Business Assurance and GFK Eurisko, 52% of enterprises globally see Big Data as an opportunity and 76% of all organizations are planning to increase or maintain their investment in big data over the next two to three years. In line with the survey, there is a growing interest from MuleSoft’s ecosystem in big data, which we are happy to support with our Anypoint Connector for Hadoop (HDFS) v5.0.0.

The Hadoop (HDFS) Connector v5.0.0 is built based on Hadoop 2.7.2 and is tested against Hadoop 2.7.1 / 2.7.2 and Hortonworks Data Platform(HDP) 2.4, which includes Hadoop 2.7.1. In this blog post, I’d like to walk you through how to use the Hadoop (HDFS) Connector v5.0.0 with a demo app called “common-operations”.

Before we start, please make sure you have access to Hadoop v2.7.1 or newer, if not, you can easily install one from the Apache Hadoop website. For the following demo, I’m going to use Hadoop 2.7.2 locally installed on Mac. After I run Hadoop 2.7.2 and hit localhost:50070, I can see the following page. (You might see a slightly different view based on your Hadoop version.)

screen-shot-2016-09-21-at-4-10-56-pm

Before you try the connector, please make sure you have Hadoop (HDFS) Connector v5.0.0 installed in Anypoint Studio. If not, please download it from the Exchange.

screen-shot-2016-09-21-at-4-22-55-pm

Once you download the common-operations demo app from this page and import it into Studio, you will see the following app showing you the CRUD operations on file and directory.

screen-shot-2016-09-21-at-4-24-46-pm

After you import the demo app, select “Global Element” and open the “HDFS: Simple Configuration” by clicking on “Edit”.

screen-shot-2016-09-21-at-4-27-13-pm

You can specify your HDFS configuration directly here, but I recommend you use the mule-app.properties. In mule-app.properties, configure the following keys:

config.nameNodeUri=hdfs://localhost:9000 (Yours can be different)
config.sysUser= (I have not set up any sysUser.)

If you start the demo app in Studio and hit localhost:8090/ with your browser, you can see the simple HTML page helping you play with operations supported by Hadoop HDFSConnector v5.0.0.

screen-shot-2016-09-21-at-4-31-34-pm

You can simply create a file with the “Create File” form. I created the hellohdfs.txt with the following information:

Path: connectordemo/hellohdfs.txt
Content: Connect anything. Change everything.

screen-shot-2016-09-21-at-4-35-02-pm

As you can see below, hellohdfs.txt is created under /connectordemo.

screen-shot-2016-09-21-at-4-36-03-pm

While you can try out other operations, I’d like to highlight a new operation called “Read from path” which we added with Hadoop HDFSConnector v5.0.0. With this new version, the connector can read the content of a file designated by its path and stream it to the rest of the flow. You don’t have to drop a poll component in source to periodically patch a file. To try this out, first, specify the path (i.e. /connectordemo/hellohdfs.txt) and change the initial state of the flow from “stopped” to “started”.

screen-shot-2016-09-21-at-4-47-02-pm

For new users, try the above example to get started, and for others, please share with us how you use or are planning to use the Hadoop HDFS Connector! Also, explore the Anypoint Exchange to see other resources you can leverage today.

hadoop Connector (mathematics) Big data app planning Download Flow (web browser) Leverage (statistics)

Published at DZone with permission of , DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

Popular on DZone

  • Choosing the Best Cloud Provider for Hosting DevOps Tools
  • Top 5 Java REST API Frameworks
  • Utilize OpenAI API to Extract Information From PDF Files
  • Deploying Java Serverless Functions as AWS Lambda

Comments

Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com
  • +1 (919) 678-0300

Let's be friends: