Over a million developers have joined DZone.

Spark Tips: Must Have For Twitter Batch Processing

Here are some tips for working with Twitter data stored in Hive for processing with Scala Spark batch jobs.

· Big Data Zone

Read this eGuide to discover the fundamental differences between iPaaS and dPaaS and how the innovative approach of dPaaS gets to the heart of today’s most pressing integration problems, brought to you in partnership with Liaison.

Here are some quick tips and code blocks for Twitter batch processing.

name := "Sentiment"version := "1.0"
scalaVersion := "2.10.6"
assemblyJarName in assembly := "sentiment.jar"
libraryDependencies += "org.apache.spark" % "spark-core_2.10" % "1.6.0" % "provided"
libraryDependencies += "org.apache.spark" % "spark-sql_2.10" % "1.6.0" % "provided"
libraryDependencies += "org.apache.spark" %% "spark-hive" % "1.6.0" % "provided"
libraryDependencies += "edu.stanford.nlp" % "stanford-corenlp" % "3.5.1"
libraryDependencies += "edu.stanford.nlp" % "stanford-corenlp" % "3.5.1" classifier "models"
resolvers += "Akka Repository" at "http://repo.akka.io/releases/"
assemblyMergeStrategy in assembly := {  
  case PathList("META-INF", xs @ _*) => MergeStrategy.discard  
  case x => MergeStrategy.first}

A Scala class to hold the most important Twitter fields from JSON Tweet.

  case class Tweet(coordinates: String, geo:String, handle: String, 
                   hashtags: String, language: String,
                   location: String, msg: String, time: String, 
                   tweet_id: String, unixtime: String, 
                   user_name: String, tag: String,
                   profile_image_url: String,
                   source: String, place: String, friends_count: String, 
                   followers_count: String, retweet_count: String, 
                   time_zone: String, sentiment: String, 
                   stanfordSentiment: String)

The imported packages that you will most likely need.

import java.util.Properties
import com.vader.SentimentAnalyzer
import edu.stanford.nlp.ling.CoreAnnotations
import edu.stanford.nlp.neural.rnn.RNNCoreAnnotations
import edu.stanford.nlp.pipeline.StanfordCoreNLP
import edu.stanford.nlp.sentiment.SentimentCoreAnnotations
import org.apache.log4j.{Level, Logger}
import org.apache.spark.{SparkConf, SparkContext}
import org.apache.spark.serializer.KryoSerializer
import org.apache.spark.sql._

A snippet of Scala Spark code to read Hive data.

ef main(args: Array[String]) {


  val logger: Logger = Logger.getLogger("com.dataflowdeveloper.sentiment.TwitterSentimentAnalysis")
  val sparkConf = new SparkConf().setAppName("TwitterSentimentAnalysis")

  sparkConf.set("spark.streaming.backpressure.enabled", "true")
  sparkConf.set("spark.cores.max", "32")
  sparkConf.set("spark.serializer", classOf[KryoSerializer].getName)
  sparkConf.set("spark.sql.tungsten.enabled", "true")
  sparkConf.set("spark.eventLog.enabled", "true")
  sparkConf.set("spark.app.id", "Sentiment")
  sparkConf.set("spark.io.compression.codec", "snappy")
  sparkConf.set("spark.rdd.compress", "true")

  val sc = new SparkContext(sparkConf)
  val sqlContext = new org.apache.spark.sql.hive.HiveContext(sc)
  import sqlContext.implicits._
  val tweets = sqlContext.read.json("hdfs://sandbox.hortonworks.com:8020/social/twitter")
  sqlContext.setConf("spark.sql.orc.filterPushdown", "true")


It's pretty straight forward but you need the Hive context, not the standard SQL context.

To make sure your data looks good, check the original JSON external table and then check any tables you may create.   I like to create a new ORC Hive table for processed data from Spark.  

!connect jdbc:hive2://localhost:10000/default;

!set showHeader true;
set hive.vectorized.execution.enabled=true;
set hive.execution.engine=tez;
set hive.vectorized.execution.enabled =true;
set hive.vectorized.execution.reduce.enabled =true;
set hive.compute.query.using.stats=true;
set hive.cbo.enable=true;
set hive.stats.fetch.column.stats=true;
set hive.stats.fetch.partition.stats=true;
show tables;
describe sparktwitterorc;
describe twitterraw;
describe sparktwitterorc;
analyze table sparktwitterorc compute statistics;
analyze table sparktwitterorc compute statistics for columns;

The twitterraw Hive table is the external table holding JSON.  The sparktwitterorc table is the ORC Hive table created by Spark.

I have included some parameters in my beeline script for improving performance, remember Tez and ORC together are super charged.

How do you write a RDD or DataFrame to a Hive ORC table????


On a memory challenged machine, you may need some options for building.

export SBT_OPTS="-Xmx2G -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=2G -Xss2M  -Duser.timezone=GMT"
sbt -J-Xmx4G -J-Xms4G assembly

Submitting your Spark Scala job to a Spark 1.6.x server on YARN is uber easy:

spark-submit --class com.dataflowdeveloper.sentiment.TwitterSentimentAnalysis --master yarn-client sentiment.jar --verbose 
An example Hive Table:

CREATE TABLE rawtwitter
   handle              STRING,
   hashtags            STRING,
   msg                 STRING,
   language            STRING,
   time                STRING,
   tweet_id            STRING,
   unixtime            STRING,
   user_name           STRING,
   geo                 STRING,
   coordinates         STRING,
   `location`          STRING,
   time_zone           STRING,
   retweet_count       STRING,
   followers_count     STRING,
   friends_count       STRING,
   place               STRING,
   source              STRING,
   profile_image_url   STRING,
   tag                 STRING,
   sentiment           STRING,
   stanfordsentiment   STRING
ROW FORMAT SERDE 'org.apache.hive.hcatalog.data.JsonSerDe'
LOCATION 'hdfs://sandbox.hortonworks.com:8020/social/twitter'

Image title

Discover the unprecedented possibilities and challenges, created by today’s fast paced data climate and why your current integration solution is not enough, brought to you in partnership with Liaison

spark,scala,hadoop,big data,hortonworks,twitter,social media,hive,sql

The best of DZone straight to your inbox.

Please provide a valid email address.

Thanks for subscribing!

Awesome! Check your inbox to verify your email so you can start receiving the latest in tech news and resources.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}