Getting to Know Amazon Elastic MapReduce
Join the DZone community and get the full member experience.Join For Free
amazon elastic mapreduce is a service in the aws portfolio that can be used for data processing and analytics on vast amounts of data. it is based on hadoop (as of writing this article it is using hadoop 0.20.205) and relies on other aws services such as ec2 and s3.
the data processing applications can be implemented using various technologies such as hive, pig, java (custom jar) and streaming (e.g. python or ruby). this post will demonstrate how to use hive on amazon elastic mapreduce – the sample application will calculate the average price of apple stock in every year from 1984 till 2012. at the time of writing hive version is 0.7.1 . (side note: as it will be shown, aapl started at around 25 usd as an average price in 1984, managed to get down to 18 usd in 1997 and now it is around 500 – 496.32138, to be more precise -, quite some numbers for a company that is in infinite loop for decades…)
how to create elastic mapreduce jobs?
there are three steps to manage an emr jobflow:
1./ upload the script (i.e. hive.q file) and the data to be processed onto s3. if you are unfamiliar with aws, this is a good place to start to understand its structure and the way how to use it.
the test data used in the post is downloaded from yahoo! finance website (historical data for aapl stock). go to http://finance.yahoo.com/q/hp?s=aapl+historical+prices and then scroll down to download to spreadsheet link. this will create a csv file (~6,950 lines) with the following columns: date,open,high,low,close,volume,adj close. remove the header (the first line) to leave only the relevant data in the csv file.
steps to upload the input files:
a./ go to aws s3 console and create “stockprice” bucket:
b./ create folders under stockprice bucket: apple/input, apple/output and hive-scripts.
c./ upload apple.q hive-script into //stockprice/hive-scripts folder
d./ upload the csv input file containing aapl stock prices into //stockprice/apple/input folder
2./ create an elastic mapreduce jobflow:
natigate to https://console.aws.amazon.com/elasticmapreduce/home
b./ select “create new job flow”
c./ configure job parameters:
d./ configure ec2 instances:
e./ define ec2 key pair:
f./ if you want, you can configure debugging by defining a s3 log path and selecting “enable debugging” (optional). i highly recommend to do it if you are in development phase:
g./ set no bootstrap actions:
h./ review the configuration before you hit the run button:
i./ create job flow:
j./ you can verify the job flow status from starting to running to shutdown.
should there be any issues occuring, you can check the stderr, stdout, syslog from “debug” menu.
3./ check the result:
after a few minutes of number crunching, the output will be generated in //stokcprice/apple/output folder (e.g. 000000 file). the file will have a text format with date and stock price cloumns (separeted by soh – start of heading – ascii 001), see:
the hive code to process the data (apple.q) looks like this:
create external table stockprice ( yyyymmdd string, open_price float, high_price float, low_price float, close_price float, stock_volume int, adjclose_price float ) row format delimited fields terminated by ‘,’ lines terminated by ‘\n’ location ‘s3://stockprice/apple/input/’; create table tmp_stockprice ( year int, close_price float ) stored as sequencefile; insert overwrite table tmp_stockprice select year(sp.yyyymmdd), sp.close_price from stockprice sp; create table avg_yearly_stockprice ( year int, avg_price float ) row format delimited fields terminated by ‘\t’ lines terminated by ‘\n’ stored as textfile; insert overwrite table avg_yearly_stockprice select tmp_sp.year, avg(tmp_sp.close_price) from tmp_stockprice tmp_sp group by tmp_sp.year; insert overwrite directory ‘s3://stockprice/apple/output/’ select * from avg_yearly_stockprice;
alternatively you can define location for avg_yearly_stockprice in a similar way (external table) as it is done stockprice table instead of insert overwrite directory.
Published at DZone with permission of Istvan Szegedi, DZone MVB. See the original article here.
Opinions expressed by DZone contributors are their own.