DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports Events Over 2 million developers have joined DZone. Join Today! Thanks for visiting DZone today,
Edit Profile Manage Email Subscriptions Moderation Admin Console How to Post to DZone Article Submission Guidelines
View Profile
Sign Out
Refcards
Trend Reports
Events
Zones
Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Partner Zones AWS Cloud
by AWS Developer Relations
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Partner Zones
AWS Cloud
by AWS Developer Relations
Building Scalable Real-Time Apps with AstraDB and Vaadin
Register Now

Trending

  • Automating the Migration From JS to TS for the ZK Framework
  • Java String Templates Today
  • API Governance: Best Practices and Strategies for Effective API Management
  • What to Pay Attention to as Automation Upends the Developer Experience

Trending

  • Automating the Migration From JS to TS for the ZK Framework
  • Java String Templates Today
  • API Governance: Best Practices and Strategies for Effective API Management
  • What to Pay Attention to as Automation Upends the Developer Experience
  1. DZone
  2. Data Engineering
  3. Big Data
  4. Unit Testing a Java Hadoop Job

Unit Testing a Java Hadoop Job

$$anonymous$$ user avatar by
$$anonymous$$
·
Aug. 29, 13 · Interview
Like (2)
Save
Tweet
Share
9.77K Views

Join the DZone community and get the full member experience.

Join For Free

in my previous post i showed how to setup a complete maven based project to create a hadoop job in java. of course it wasn’t complete because it is missing the unit test part . in this post i show how to add mapreduce unit tests to the project i started previously. for the unit test i make use of the mrunit framework .

  • add the necessary dependency to the pom
  • add the following dependency to the pom:

    <dependency>
    <groupid>org.apache.mrunit</groupid>
    <artifactid>mrunit</artifactid>
    <version>1.0.0</version>
    <classifier>hadoop1</classifier>
    <scope>test</scope>
    </dependency>


    this will made the mrunit framework available to the project.

  • add unit tests for testing the map reduce logic
  • the use of this framework is quite straightforward, especially in our business case. so i will just show the unit test code and some comments if necessary but i think it is quite obvious how to use it.
    the unit test for the mapper ‘mappertest’:

    package net.pascalalma.hadoop;
    import org.apache.hadoop.io.text;
    import org.apache.hadoop.mrunit.mapreduce.mapdriver;
    import org.junit.before;
    import org.junit.test;
    import java.io.ioexception;
    /**
    * created with intellij idea.
    * user: pascal
    */
    public class mappertest {
    mapdriver<text, text, text, text> mapdriver;
    @before
    public void setup() {
    wordmapper mapper = new wordmapper();
    mapdriver = mapdriver.newmapdriver(mapper);
    }
    @test
    public void testmapper() throws ioexception {
    mapdriver.withinput(new text("a"), new text("ein"));
    mapdriver.withinput(new text("a"), new text("zwei"));
    mapdriver.withinput(new text("c"), new text("drei"));
    mapdriver.withoutput(new text("a"), new text("ein"));
    mapdriver.withoutput(new text("a"), new text("zwei"));
    mapdriver.withoutput(new text("c"), new text("drei"));
    mapdriver.runtest();
    }
    }

    this test class is actually even simpler than the mapper implementation itself. you just define the input of the mapper and the expected output and then let the configured mapdriver run the test. in our case the mapper doesn’t do anything specific but you see how easy it is to setup a testcase.
    for completeness here is the test class of the reducer:

    package net.pascalalma.hadoop;
    import org.apache.hadoop.io.text;
    import org.apache.hadoop.mrunit.mapreduce.reducedriver;
    import org.junit.before;
    import org.junit.test;
    import java.io.ioexception;
    import java.util.arraylist;
    import java.util.list;
    /**
    * created with intellij idea.
    * user: pascal
    */
    public class reducertest {
    reducedriver<text, text, text, text> reducedriver;
    @before
    public void setup() {
    alltranslationsreducer reducer = new alltranslationsreducer();
    reducedriver = reducedriver.newreducedriver(reducer);
    }
    @test
    public void testreducer() throws ioexception {
    list<text> values = new arraylist<text>();
    values.add(new text("ein"));
    values.add(new text("zwei"));
    reducedriver.withinput(new text("a"), values);
    reducedriver.withoutput(new text("a"), new text("|ein|zwei"));
    reducedriver.runtest();
    }
    }
  • run the unit tests it
  • with the maven command “mvn clean test” we can run the tests:
    screen shot 2013-08-23 at 20.12.50

with the unit tests in place i would say we are ready to build the project and deploy it to an hadoop cluster, which i will describe in the next post.

unit test hadoop career Java (programming language)

Published at DZone with permission of $$anonymous$$. See the original article here.

Opinions expressed by DZone contributors are their own.

Trending

  • Automating the Migration From JS to TS for the ZK Framework
  • Java String Templates Today
  • API Governance: Best Practices and Strategies for Effective API Management
  • What to Pay Attention to as Automation Upends the Developer Experience

Comments

Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com

Let's be friends: