Installing Apache Mesos 1.6.0 on Ubuntu 18.04

In this post, I will show you how to install Mesos on Ubuntu 18.04. This article is part of my new series on Fast Data Architecture. I have installed Mesos on other versions of Ubuntu but had some difficulties getting everything working on Ubuntu 18.04. We will overcome each one and get Mesos 1.6.0 installed.

Cluster Design

This cluster will be similar to other Mesos/Marathon clusters I have created. I will make this one a little beefier because I want to use it as a training platform for the Fast Data Architecture using a SMACK stack. I will go more in-depth on what a SMACK stack is in a later post. But, to sum up, it is a Fast Data Architecture that utilizes Spark, Mesos, Akka, Cassandra, and Kafka to implement one form of Big Data. You can find out more from this free ebook from the folks at Mesosphere which inspired this series of posts.

I built 5 virtual machines. Three of them are going to be Mesos Masters and have 4 vCPUs and 4 GB of memory. I made them this big so that when I compiled Mesos it would not take all night. The last two virtual machines are the Mesos Slaves and I made them pretty stout at 8 vCPUs and 8 GB of memory each. I knew I needed a little horsepower to run my little lab so these guys are pretty strong (for my lab environment that is).

Here are the systems including hostnames and IP configurations.

Mesos Master 1 - mesos1.admintome.lab  192.168.1.30
Mesos Master 2 - mesos2.admintome.lab  192.168.1.31
Mesos Master 3 - mesos3.admintome.lab  192.168.1.32

Mesos Slave 1 - mslave1.admintome.lab 192.168.1.33
Mesos Slave 2 - mslave2.admintome.lab 192.168.1.34

All systems have a static IP address and I set up my local DNS servers with the hostnames so that NS lookups on the systems would point to the correct IP. If you don't have local DNS servers then simply update your /etc/hosts file to look like this on each server:

192.168.1.30  mesos1.admintome.lab   mesos1
192.168.1.31  mesos2.admintome.lab   mesos2
192.168.1.32  mesos3.admintome.lab   mesos3
192.168.1.33  mslave1.admintome.lab  mslave1
192.168.1.34  mslave2.admintome.lab  mslave2

You will of course have to change the domain to match yours or if you don't have a domain exclude that second entry on each line. Enough of the boring stuff, time to get started configuring Mesos on Ubuntu 18.04!

Install Mesos on Ubuntu 18.04

You will need to run the commands in this section on all your virtual machines including the Mesos Slaves.

As with most of my how-tos, we need to first update and upgrade a fresh install of Ubuntu 18.04.

# apt update && apt upgrade -y

Reboot your system and we can begin installing the dependencies.

# apt install openjdk-8-jdk -y
# export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/
# apt -y install build-essential python-dev libcurl4-nss-dev libsasl2-dev libsasl2-modules maven libapr1-dev libsvn-dev zlib1g-dev
# apt install unzip

I discovered that there is a bug with Ubuntu 18.04 and Java that causes an issue with Maven during the compile and shows an error similar to this:

java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty

I beat my head on the wall for two hours trying to find the fix until I came across this awesome StackOverflow answer that fixes the issue. Run these commands and you won't have the same problem:

# /usr/bin/printf '\xfe\xed\xfe\xed\x00\x00\x00\x02\x00\x00\x00\x00\xe2\x68\x6e\x45\xfb\x43\xdf\xa4\xd9\x92\xdd\x41\xce\xb6\xb2\x1c\x63\x30\xd7\x92' > /etc/ssl/certs/java/cacerts
# /var/lib/dpkg/info/ca-certificates-java.postinst configure

Your systems are now set up to compile everything successfully.

Compile the Source Code

I found that this works best if you run this section on one server at a time. We will compile Mesos on each and every node in the cluster. Which sucks because it takes forever. I don't know if you can compile on one server and install on the remaining servers so I asked a question on StackOverflow. Hopefully, the experts will show me what I am doing wrong and make this install a lot quicker.

The first thing we have to do is download the source for Apache Mesos 1.6.0:

# wgethttp://www.apache.org/dist/mesos/1.6.0/mesos-1.6.0.tar.gz

Unzip the tarball and configure the build:

# tar -xzvf mesos-1.6.0.tar.gz
# cd mesos-1.6.0
# mkdir build
# cd build

# ../configure

Now we need to start compiling the source code. The Apache Mesos page says to use the number of cores when setting the -j parameter. I have found that when I did this the virtual machines would run out of memory no matter how much I gave it and would stop compiling. They would just freeze. This cost me a couple of nights of pain. It turns out that what worked for me was to make this number equal to half the number of cores your virtual machine has. Unless, of course, your virtual machine only has one core then set the value to 1. In this example, I was compiling on a virtual machine that had 4 cores so I set the value to 2:

# make -j 2 V=0

Sit back and relax (read some other great articles from the AdminTome Blog) because this is gonna take a while. Once it is complete, you can install the binaries with these commands:

# make install
# ldconfig

That completes the source compile section. Now move on to your next Mesos server and repeat this step. Once you have the source code compiled on all your servers move on to the next section.

Configuring the Mesos Masters

This section will walk you through configuring your Mesos Master servers only. Our first step is to get Apache Zookeeper installed. Apache Zookeeper is a distributed key/value store.

Installing Apache Zookeeper

We will use the built-in canonical deployment of Zookeeper to keep things simple.

# apt install zookeeperd -y

To configure Zookeeper, we need to edit a couple of files. First, open your favorite editor and edit /etc/zookeeper/conf/myid. This will be a simple number that represents the ID of the Zookeeper node. Since I have three nodes, I numbered them 1 through 3. For example, on my mesos1.admintome.lab system I have '1' as the contents of /etc/zookeeper/conf/myid.

Next, edit /etc/zookeeper/conf/zoo.cfg. Find the three files that look like this:

#server.1=zookeeper1:2888:3888
#server.2=zookeeper2:2888:3888
#server.3=zookeeper3:2888:3888

Uncomment them and change them to reflect your three Mesos Masters.

server.1=192.168.1.30:2888:3888
server.2=192.168.1.31:2888:3888
server.3=192.168.1.32:2888:3888

Save and exit the file. We can start and enable the Zookeeper service.

# systemctl start zookeeper
# systemctl enable zookeeper

Complete this step on all three of your Mesos Master servers before continuing

SystemD Service for Apache Mesos Masters

Make a new directory

# mkdir /etc/mesos-master

Next, we need to make a /etc/mesos-master/zk file:

zk://192.18.1.30:2181,192.168.1.31:2181,192.168.1.32:2181/mesos

For the zk parameter, make sure you keep the port after each IP ':2181' because it has to be there. Save and exit the file.

Open up your favorite editor again and create a new file, /etc/systemd/system/mesos-master.service, with the following content:

[Unit]
Description=Mesos Master Service
After=zookeeper.service
Requires=zookeeper.service

[Service]
ExecStart=/usr/local/sbin/mesos-master --ip=192.168.1.30 --work_dir=/var/lib/mesos --zk=file://etc/mesos-master/zk --quorum=2 --cluster=admintome

[Install]
WantedBy=multi-user.target

Make sure all the IPs match your three Mesos Master servers. Also, make sure that the -ip parameter matches the IP address for the current master server you are configuring. The last parameter is just what the name of the cluster will be when we look at the Web UI for Apache Mesos. You can make it anything you want. Save the file and exit.

Start and Enable the Service

Time to start our new Apache Mesos SystemD service.

# systemctl daemon-reload
# systemctl start mesos-master.service
# systemctl enable mesos-master

Complete this step on all three of your Mesos Masters. Your masters are now configured and ready to go. To verify that everything is working correctly so far you can go to the following URL.

http://{mesos master IP}:5050

You can pick any one of your Mesos Master IP's here. When Mesos starts up on all three servers it will use Zookeeper to pick a master that will handle all the work. If you browse to a server that was not elected as the master you will be automatically redirected to the master server.

You should see the Mesos Web GUI now:

Pat yourself on the back, you are halfway through installing Mesos on Ubuntu 18.04. Now on to configuring those Mesos Slaves.

Configuring the Mesos Slaves

You should already have Mesos compiled on your slaves from the beginning of this post. Now, all we have to do is create our SystemD service on each one of the slaves.

Make a new directory:

# mkdir /etc/mesos-slave

Next, we need to make a /etc/mesos-slave/master file:

zk://192.18.1.30:2181,192.168.1.31:2181,192.168.1.32:2181/mesos

For the zk parameter, make sure you keep the port after each IP as ':2181' because it has to be there. Save and exit the file.

Open up your favorite editor again and create a new file, /etc/systemd/system/mesos-slave.service, with the following content:

[Unit]
Description=Mesos Slave Service

[Service]
ExecStart=/usr/local/sbin/mesos-slave --master=file://etc/mesos-slave/master --work_dir=/var/lib/mesos

[Install]
WantedBy=multi-user.target

Save and exit.

Start and Enable the Service

Time to start our new Apache Mesos Slave SystemD service.

# systemctl daemon-reload
# systemctl start mesos-slave.service
# systemctl enable mesos-slave

Complete this step on both of your Mesos Slaves.

To verify that they are both working you can view the Mesos Web UI again. You should see two activated agents like below:

As you can see here, we have 2 activated nodes and, under resources, we can see that we have 8 CPUs, ~6GB of memory and 200GB of disk space! Our two slaves are up and operational.

Celebrate!

Congratulations! You made it through and now have a complete Apache Mesos cluster installed on Ubuntu 18.04! It was a long process but well worth the time if you are studying big data. This was a ton to go through - if you wanted to set up a production cluster, however, this wouldn't be recommended. In past versions, I used configuration management systems like Puppet to configure the cluster. This wasn't an option at this time because Puppet Enterprise doesn't have a puppet agent on Ubuntu 18.04. I'm sure that will change soon so keep checking back for a post on how to setup Mesos using Puppet on Ubuntu 18.04. Thanks for sticking with me through this post, I know it was a long one but one. I have been itching to write this ever since Ubuntu 18.04 was released.

Conclusion

I hope you have enjoyed this article. If so, please leave a comment below. Thanks again for reading this post!