How to Quickly Launch a Cassandra Cluster on Amazon EC2

if you have read my previous post, “ map-reduce with ruby using hadoop “, then you will know that firing up a hadoop cluster is really simple when you use whirr. without even ssh’ing on the machines in the cloud you can start-up your cluster and interact with it. in this post i’ll show you that it is just as easy to fire up a cassandra cluster on amazon ec2.

install whirr

i will fly through the setup of whirr quite quickly. all the commands you need are here, but if you want a more thorough explanation then see my other post, “ map-reduce with ruby using hadoop “.

i am assuming that you have homebrew installed .

sudo brew update
sudo brew install maven
mkdir ~/src/cloudera
cd ~/src/cloudera
wget http://archive.cloudera.com/cdh/3/whirr-0.1.0+23.tar.gz
tar -xvzf whirr-0.1.0+23.tar.gz
cd whirr-0.1.0+23
mvn clean install
mvn package -ppackage

be patient with the above. there is a lot to install, so it will take some time. maven installs a lot of dependencies if it is your first time using it.

the good news is that from here on you are setup to easily fire-up your amazon ec2 cluster for cassandra, or hadoop if you choose.

whirr configuration file

we will need to make a configuration file for whirr to tell it that we want to launch a cassandra cluster with 3 nodes. if you are brave, patient and have the cash, then you could just as easily fire-up a 100 node cluster (leave a comment if you do – there may be prizes!).

you will need to create a cassandra.properties file with the following contents…

whirr.service-name=cassandra
whirr.cluster-name=mycassandracluster
whirr.instance-templates=3 cassandra
whirr.provider=ec2
whirr.identity=<your_amazon_ec2_access_key_id_goes_here>
whirr.credential=<your_amazon_ec2_secret_access_key_goes_here>
whirr.private-key-file=${sys:user.home}/.ssh/id_rsa

replace the obvious fields with your amazon ec2 access key id and amazon ec2 secret access key.

launch your cluster

now you are ready to fire-up your cassandra cluster. simply use the following command and then be prepared to wait 5-10 minutes while amazon builds your machines. this time is variable. sometimes amazon is quick, sometimes not so quick.

bin/whirr launch-cluster --config cassandra.properties


launching mycassandracluster cluster
configuring template
starting 3 node(s)
nodes started: [[id=us-east-1/i-13f25e7f, providerid=i-13f25e7f, tag=mycassandracluster, name=null, location=[id=us-east-1d, scope=zone, description=us-east-1d, parent=us-east-1], uri=null, imageid=us-east-1/ami-74f0061d, os=[name=null, family=amzn-linux, version=2010.11.1-beta, arch=paravirtual, is64bit=true, description=amazon/amzn-ami-2010.11.1-beta.x86_64-ebs], usermetadata={}, state=running, privateaddresses=[10.204.99.163], publicaddresses=[50.16.155.106], hardware=[id=t1.micro, providerid=t1.micro, name=t1.micro, processors=[[cores=1.0, speed=1.0]], ram=630, volumes=[[id=vol-1657d47e, type=san, size=null, device=/dev/sda1, durable=true, isbootdevice=true]], supportsimage=hasrootdevicetype(ebs)]], [id=us-east-1/i-17f25e7b, providerid=i-17f25e7b, tag=mycassandracluster, name=null, location=[id=us-east-1d, scope=zone, description=us-east-1d, parent=us-east-1], uri=null, imageid=us-east-1/ami-74f0061d, os=[name=null, family=amzn-linux, version=2010.11.1-beta, arch=paravirtual, is64bit=true, description=amazon/amzn-ami-2010.11.1-beta.x86_64-ebs], usermetadata={}, state=running, privateaddresses=[10.117.43.129], publicaddresses=[50.16.85.79], hardware=[id=t1.micro, providerid=t1.micro, name=t1.micro, processors=[[cores=1.0, speed=1.0]], ram=630, volumes=[[id=vol-1457d47c, type=san, size=null, device=/dev/sda1, durable=true, isbootdevice=true]], supportsimage=hasrootdevicetype(ebs)]], [id=us-east-1/i-11f25e7d, providerid=i-11f25e7d, tag=mycassandracluster, name=null, location=[id=us-east-1d, scope=zone, description=us-east-1d, parent=us-east-1], uri=null, imageid=us-east-1/ami-74f0061d, os=[name=null, family=amzn-linux, version=2010.11.1-beta, arch=paravirtual, is64bit=true, description=amazon/amzn-ami-2010.11.1-beta.x86_64-ebs], usermetadata={}, state=running, privateaddresses=[10.117.46.170], publicaddresses=[184.73.100.203], hardware=[id=t1.micro, providerid=t1.micro, name=t1.micro, processors=[[cores=1.0, speed=1.0]], ram=630, volumes=[[id=vol-e857d480, type=san, size=null, device=/dev/sda1, durable=true, isbootdevice=true]], supportsimage=hasrootdevicetype(ebs)]]]
authorizing firewall
running configuration script
completed launch of mycassandracluster
started cluster of 3 instances
cluster{instances=[instance{roles=[cassandra], publicaddress=/50.16.85.79, privateaddress=/10.117.43.129}, instance{roles=[cassandra], publicaddress=/50.16.155.106, privateaddress=/10.204.99.163}, instance{roles=[cassandra], publicaddress=/184.73.100.203, privateaddress=/10.117.46.170}], configuration={}}

you now have your very own cassandra cluster running in the cloud. not so hard, hey!

connect from ruby

i will be following this post with step-by-step guide on how you can interact with your new cluster from your ruby on rails application. i recommend subscribing to the rss feed to get updates to the blog.

shutdown the cluster

here is how you can shutdown your cluster.

bin/whirr destroy-cluster --config cassandra.properties

destroying mycassandracluster cluster
cluster mycassandracluster destroyed

conclusion

whirr makes it very easy to start and stop a cassandra cluster in the cloud without leaving the comfort of your laptop. what you do with that cluster is up to you, but i will be give you some ideas of what you could do in future posts.