Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

Word Count Programs: Spark vs. MapReduce

DZone's Guide to

Word Count Programs: Spark vs. MapReduce

Why use Spark for big data processing? Compare these two word count program examples.

· Big Data Zone
Free Resource

Effortlessly power IoT, predictive analytics, and machine learning applications with an elastic, resilient data infrastructure. Learn how with Mesosphere DC/OS.

Apache Spark is a powerful data processing tool in the distributed computing arena.

Compared to Hadoop, it is 10x faster on disk and 100x faster in memory.

The only complexity with Apache Spark is that it is built on Scala, so there is a new learning curve involved, but take my word for it: it is very easy and if you are a developer from a Java background you will really like it. :)

Today I will compare simple Word Count examples which are implemented by using both MapReduce and Spark.

Word Count Example (MapReduce)

     package org.myorg;

     import java.io.IOException;
     import java.util.*;

     import org.apache.hadoop.fs.Path;
     import org.apache.hadoop.conf.*;
     import org.apache.hadoop.io.*;
     import org.apache.hadoop.mapred.*;
     import org.apache.hadoop.util.*;

     public class WordCount {

        public static class Map extends MapReduceBase implements Mapper<LongWritable, Text, Text, IntWritable> {
          private final static IntWritable one = new IntWritable(1);
          private Text word = new Text();

          public void map(LongWritable key, Text value, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException {
            String line = value.toString();
            StringTokenizer tokenizer = new StringTokenizer(line);
            while (tokenizer.hasMoreTokens()) {
              word.set(tokenizer.nextToken());
              output.collect(word, one);
            }
          }
        }

        public static class Reduce extends MapReduceBase implements Reducer<Text, IntWritable, Text, IntWritable> {
          public void reduce(Text key, Iterator values, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException {
            int sum = 0;
            while (values.hasNext()) {
              sum += values.next().get();
            }
            output.collect(key, new IntWritable(sum));
          }
        }

        public static void main(String[] args) throws Exception {
          JobConf conf = new JobConf(WordCount.class);
          conf.setJobName("wordcount");

          conf.setOutputKeyClass(Text.class);
          conf.setOutputValueClass(IntWritable.class);

          conf.setMapperClass(Map.class);
          conf.setCombinerClass(Reduce.class);
          conf.setReducerClass(Reduce.class);

          conf.setInputFormat(TextInputFormat.class);
          conf.setOutputFormat(TextOutputFormat.class);

          FileInputFormat.setInputPaths(conf, new Path(args[0]));
          FileOutputFormat.setOutputPath(conf, new Path(args[1]));

          JobClient.runJob(conf);
        }
     }

Word Count (Using Spark)

 val file = spark.textFile("hdfs://...")

 val counts = file.flatMap(line => line.split(" ")).map(word => (word, 1)).reduceByKey(_ + _)

 counts.saveAsTextFile("hdfs://...")

If you notice, it took 58 lines to implement WordCount program using MapReduce Paradigm but the same WordCount was just implemented in 3 lines using Spark.

So Spark is a really powerful data processing tool both from a latency and readability point of view.

Hope this article helps in giving clarity to users on why you should use Spark for big data processing.

Learn to design and build better data-rich applications with this free eBook from O’Reilly. Brought to you by Mesosphere DC/OS.

Topics:
big data ,spark ,mapreduce ,scala ,java

Opinions expressed by DZone contributors are their own.

THE DZONE NEWSLETTER

Dev Resources & Solutions Straight to Your Inbox

Thanks for subscribing!

Awesome! Check your inbox to verify your email so you can start receiving the latest in tech news and resources.

X

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}