Spark Tips: Must Have For Twitter Batch Processing
Here are some tips for working with Twitter data stored in Hive for processing with Scala Spark batch jobs.
Join the DZone community and get the full member experience.
Join For FreeHere are some quick tips and code blocks for Twitter batch processing.
name := "Sentiment"version := "1.0"
scalaVersion := "2.10.6"
assemblyJarName in assembly := "sentiment.jar"
libraryDependencies += "org.apache.spark" % "spark-core_2.10" % "1.6.0" % "provided"
libraryDependencies += "org.apache.spark" % "spark-sql_2.10" % "1.6.0" % "provided"
libraryDependencies += "org.apache.spark" %% "spark-hive" % "1.6.0" % "provided"
libraryDependencies += "edu.stanford.nlp" % "stanford-corenlp" % "3.5.1"
libraryDependencies += "edu.stanford.nlp" % "stanford-corenlp" % "3.5.1" classifier "models"
resolvers += "Akka Repository" at "http://repo.akka.io/releases/"
assemblyMergeStrategy in assembly := {
case PathList("META-INF", xs @ _*) => MergeStrategy.discard
case x => MergeStrategy.first}
A Scala class to hold the most important Twitter fields from JSON Tweet.
case class Tweet(coordinates: String, geo:String, handle: String,
hashtags: String, language: String,
location: String, msg: String, time: String,
tweet_id: String, unixtime: String,
user_name: String, tag: String,
profile_image_url: String,
source: String, place: String, friends_count: String,
followers_count: String, retweet_count: String,
time_zone: String, sentiment: String,
stanfordSentiment: String)
The imported packages that you will most likely need.
import java.util.Properties
import com.vader.SentimentAnalyzer
import edu.stanford.nlp.ling.CoreAnnotations
import edu.stanford.nlp.neural.rnn.RNNCoreAnnotations
import edu.stanford.nlp.pipeline.StanfordCoreNLP
import edu.stanford.nlp.sentiment.SentimentCoreAnnotations
import org.apache.log4j.{Level, Logger}
import org.apache.spark.{SparkConf, SparkContext}
import org.apache.spark.serializer.KryoSerializer
import org.apache.spark.sql._
A snippet of Scala Spark code to read Hive data.
ef main(args: Array[String]) {
Logger.getLogger("org.apache.spark").setLevel(Level.ERROR)
Logger.getLogger("org.apache.spark.storage.BlockManager").setLevel(Level.ERROR)
val logger: Logger = Logger.getLogger("com.dataflowdeveloper.sentiment.TwitterSentimentAnalysis")
val sparkConf = new SparkConf().setAppName("TwitterSentimentAnalysis")
sparkConf.set("spark.streaming.backpressure.enabled", "true")
sparkConf.set("spark.cores.max", "32")
sparkConf.set("spark.serializer", classOf[KryoSerializer].getName)
sparkConf.set("spark.sql.tungsten.enabled", "true")
sparkConf.set("spark.eventLog.enabled", "true")
sparkConf.set("spark.app.id", "Sentiment")
sparkConf.set("spark.io.compression.codec", "snappy")
sparkConf.set("spark.rdd.compress", "true")
val sc = new SparkContext(sparkConf)
val sqlContext = new org.apache.spark.sql.hive.HiveContext(sc)
import sqlContext.implicits._
val tweets = sqlContext.read.json("hdfs://sandbox.hortonworks.com:8020/social/twitter")
sqlContext.setConf("spark.sql.orc.filterPushdown", "true")
tweets.printSchema()
tweets.count
tweets.take(5).foreach(println)
It's pretty straight forward but you need the Hive context, not the standard SQL context.
To make sure your data looks good, check the original JSON external table and then check any tables you may create. I like to create a new ORC Hive table for processed data from Spark.
beeline
!connect jdbc:hive2://localhost:10000/default;
!set showHeader true;
set hive.vectorized.execution.enabled=true;
set hive.execution.engine=tez;
set hive.vectorized.execution.enabled =true;
set hive.vectorized.execution.reduce.enabled =true;
set hive.compute.query.using.stats=true;
set hive.cbo.enable=true;
set hive.stats.fetch.column.stats=true;
set hive.stats.fetch.partition.stats=true;
show tables;
describe sparktwitterorc;
describe twitterraw;
describe sparktwitterorc;
analyze table sparktwitterorc compute statistics;
analyze table sparktwitterorc compute statistics for columns;
The twitterraw Hive table is the external table holding JSON. The sparktwitterorc table is the ORC Hive table created by Spark.
I have included some parameters in my beeline script for improving performance, remember Tez and ORC together are super charged.
How do you write a RDD or DataFrame to a Hive ORC table????
outputTweets.toDF().write.format("orc").mode(SaveMode.Overwrite).saveAsTable("default.sparktwitterorc")
On a memory challenged machine, you may need some options for building.
export SBT_OPTS="-Xmx2G -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=2G -Xss2M -Duser.timezone=GMT"
sbt -J-Xmx4G -J-Xms4G assembly
Submitting your Spark Scala job to a Spark 1.6.x server on YARN is uber easy:
spark-submit --class com.dataflowdeveloper.sentiment.TwitterSentimentAnalysis --master yarn-client sentiment.jar --verbose
An example Hive Table:
CREATE TABLE rawtwitter
(
handle STRING,
hashtags STRING,
msg STRING,
language STRING,
time STRING,
tweet_id STRING,
unixtime STRING,
user_name STRING,
geo STRING,
coordinates STRING,
`location` STRING,
time_zone STRING,
retweet_count STRING,
followers_count STRING,
friends_count STRING,
place STRING,
source STRING,
profile_image_url STRING,
tag STRING,
sentiment STRING,
stanfordsentiment STRING
)
ROW FORMAT SERDE 'org.apache.hive.hcatalog.data.JsonSerDe'
LOCATION 'hdfs://sandbox.hortonworks.com:8020/social/twitter'
Opinions expressed by DZone contributors are their own.
Comments