Spark Tips: Must Have For Twitter Batch Processing

Here are some quick tips and code blocks for Twitter batch processing.

name := "Sentiment"version := "1.0"
scalaVersion := "2.10.6"
assemblyJarName in assembly := "sentiment.jar"
libraryDependencies += "org.apache.spark" % "spark-core_2.10" % "1.6.0" % "provided"
libraryDependencies += "org.apache.spark" % "spark-sql_2.10" % "1.6.0" % "provided"
libraryDependencies += "org.apache.spark" %% "spark-hive" % "1.6.0" % "provided"
libraryDependencies += "edu.stanford.nlp" % "stanford-corenlp" % "3.5.1"
libraryDependencies += "edu.stanford.nlp" % "stanford-corenlp" % "3.5.1" classifier "models"
resolvers += "Akka Repository" at "http://repo.akka.io/releases/"
assemblyMergeStrategy in assembly := {  
  case PathList("META-INF", xs @ _*) => MergeStrategy.discard  
  case x => MergeStrategy.first}

A Scala class to hold the most important Twitter fields from JSON Tweet.

  case class Tweet(coordinates: String, geo:String, handle: String, 
                   hashtags: String, language: String,
                   location: String, msg: String, time: String, 
                   tweet_id: String, unixtime: String, 
                   user_name: String, tag: String,
                   profile_image_url: String,
                   source: String, place: String, friends_count: String, 
                   followers_count: String, retweet_count: String, 
                   time_zone: String, sentiment: String, 
                   stanfordSentiment: String)

The imported packages that you will most likely need.

import java.util.Properties
import com.vader.SentimentAnalyzer
import edu.stanford.nlp.ling.CoreAnnotations
import edu.stanford.nlp.neural.rnn.RNNCoreAnnotations
import edu.stanford.nlp.pipeline.StanfordCoreNLP
import edu.stanford.nlp.sentiment.SentimentCoreAnnotations
import org.apache.log4j.{Level, Logger}
import org.apache.spark.{SparkConf, SparkContext}
import org.apache.spark.serializer.KryoSerializer
import org.apache.spark.sql._

A snippet of Scala Spark code to read Hive data.

ef main(args: Array[String]) {

  Logger.getLogger("org.apache.spark").setLevel(Level.ERROR)
  Logger.getLogger("org.apache.spark.storage.BlockManager").setLevel(Level.ERROR)

  val logger: Logger = Logger.getLogger("com.dataflowdeveloper.sentiment.TwitterSentimentAnalysis")
  val sparkConf = new SparkConf().setAppName("TwitterSentimentAnalysis")

  sparkConf.set("spark.streaming.backpressure.enabled", "true")
  sparkConf.set("spark.cores.max", "32")
  sparkConf.set("spark.serializer", classOf[KryoSerializer].getName)
  sparkConf.set("spark.sql.tungsten.enabled", "true")
  sparkConf.set("spark.eventLog.enabled", "true")
  sparkConf.set("spark.app.id", "Sentiment")
  sparkConf.set("spark.io.compression.codec", "snappy")
  sparkConf.set("spark.rdd.compress", "true")

  val sc = new SparkContext(sparkConf)
  val sqlContext = new org.apache.spark.sql.hive.HiveContext(sc)
  import sqlContext.implicits._
  val tweets = sqlContext.read.json("hdfs://sandbox.hortonworks.com:8020/social/twitter")
  sqlContext.setConf("spark.sql.orc.filterPushdown", "true")

  tweets.printSchema()
  tweets.count
  tweets.take(5).foreach(println)

It's pretty straight forward but you need the Hive context, not the standard SQL context.

To make sure your data looks good, check the original JSON external table and then check any tables you may create.   I like to create a new ORC Hive table for processed data from Spark.  

beeline
!connect jdbc:hive2://localhost:10000/default;

!set showHeader true;
set hive.vectorized.execution.enabled=true;
set hive.execution.engine=tez;
set hive.vectorized.execution.enabled =true;
set hive.vectorized.execution.reduce.enabled =true;
set hive.compute.query.using.stats=true;
set hive.cbo.enable=true;
set hive.stats.fetch.column.stats=true;
set hive.stats.fetch.partition.stats=true;
show tables;
describe sparktwitterorc;
describe twitterraw;
describe sparktwitterorc;
analyze table sparktwitterorc compute statistics;
analyze table sparktwitterorc compute statistics for columns;

The twitterraw Hive table is the external table holding JSON.  The sparktwitterorc table is the ORC Hive table created by Spark.

I have included some parameters in my beeline script for improving performance, remember Tez and ORC together are super charged.

How do you write a RDD or DataFrame to a Hive ORC table????

outputTweets.toDF().write.format("orc").mode(SaveMode.Overwrite).saveAsTable("default.sparktwitterorc")

On a memory challenged machine, you may need some options for building.

export SBT_OPTS="-Xmx2G -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=2G -Xss2M  -Duser.timezone=GMT"
sbt -J-Xmx4G -J-Xms4G assembly

Submitting your Spark Scala job to a Spark 1.6.x server on YARN is uber easy:

spark-submit --class com.dataflowdeveloper.sentiment.TwitterSentimentAnalysis --master yarn-client sentiment.jar --verbose 
An example Hive Table:

CREATE TABLE rawtwitter
(
   handle              STRING,
   hashtags            STRING,
   msg                 STRING,
   language            STRING,
   time                STRING,
   tweet_id            STRING,
   unixtime            STRING,
   user_name           STRING,
   geo                 STRING,
   coordinates         STRING,
   `location`          STRING,
   time_zone           STRING,
   retweet_count       STRING,
   followers_count     STRING,
   friends_count       STRING,
   place               STRING,
   source              STRING,
   profile_image_url   STRING,
   tag                 STRING,
   sentiment           STRING,
   stanfordsentiment   STRING
)
ROW FORMAT SERDE 'org.apache.hive.hcatalog.data.JsonSerDe'
LOCATION 'hdfs://sandbox.hortonworks.com:8020/social/twitter'