Building Massively-Scalable Distributed Systems using Go and Mesos

DZone 's Guide to

Building Massively-Scalable Distributed Systems using Go and Mesos

· DevOps Zone ·
Free Resource

Apache Mesos is a resource manager and scheduler that operates as a system kernel for the datacenter. It abstracts CPU, memory, storage, and other compute resources away from machines (physical or virtual), enabling fault-tolerant and elastic distributed systems to easily be built and run effectively. For example, the original version of Apache Spark was built on top of Mesos when both projects were part of the AMPLab at UC Berkeley.

The Mesos API lets developers write distributed applications that take advantage of datacenter scale computing without having to think about individual hosts or write any networking code. Using the Mesos API, a cluster of even tens of thousands of machines looks like one giant machine to the developer, making it possible to write a distributed service using high-level abstractions.

Ready, Set, Go!

Apache Mesos uses an idiom known as a framework to delegate task scheduling to client code running on the cluster. Spark, for example, was originally a Mesos framework written to the Mesos API using the Scala language bindings.

The original version of Mesos was built in 2009, before Go was popular. Today, Go is one of the most popular languages and many of the key components that integrate with Mesos are written in Go. For example, Kubernetes-Mesos, the Mesos framework for running Kubernetes workloads on Mesos is written in Go. Also, Go is popular with the many infrastructure tools including and surrounding the Docker container format which is natively supported by Mesos.

It’s now possible to build Mesos frameworks in Go using pure Go language bindings. This capability will allow the Go community to build a whole new generation of distributed systems. The rest of this article will explain how the new Go bindings work in Mesos and will show you how to build a new distributed systems framework on Mesos using Go.

Mesos Framework Bindings

Prior to version 0.19.0 of Apache Mesos (introduced in June 2014), a programmer who wanted to implement a Mesos framework only had one choice to interface with the Mesos API -  mainly, using the provided libmesos.so C/C++library modulue.  This gave rise to several language bindings that rely heavily on the libmesos code to function including:

  • Java/Scala
  • Python
  • Perl
  • Haskell
  • Go

While the use of the external C/C++ module for these bindings works fine, they come with some burdensome side-effects:

  • the selected language must have a way to interface with C/C++
  • there are idiomatic gaps between the host language and the C/C++ API
  • there is a need for an abstraction layer to mitigate data representation parity issues
  • there is a build-time dependency (for compiled languages) on libmesos.so
  • libmesos.so must be distributed with code to satisfy runtime dependencies

Direct Protocol Binding

Starting with version 0.19.0, the Apache Mesos community introduced changes to the Mesos HTTP-based wire protocol that made it possible for developers to create frameworks that binds directly to the underlying HTTP stack provided by the language.  This opened the door for programmers to create Mesos frameworks purely in the language of their choice with no external dependency on C/C++ code.

Since version 0.19 was released, there have been several projects that implement the Mesos HTTP API directly in their respective native language, including:

•  Jesos, a Java implementation - https://github.com/groupon/jesos
•  Pesos, a Python Python implementation - https://github.com/wickman/pesos
•  Mesos-Go, a Go implementation (see below)

The remainder of this write up is about Mesos-Go, the pure Go implementation of the Apache-Mesos framework library.

Mesos-Go - Github.com/mesos/mesos-go

Mesos-Go is a pure implementation of the Apache Mesos framework API written in the Go programming language (http://golang.org).  As indicated earlier, the original Mesos-Go project used the C/C++ binding in its previous life.  In its latest reincarnation, however, Mesos-Go is entirely written in Go with no dependency on the C/C++ libmesos.so library module.  The project uses the HTTP wire protocol to communicate directly with a running Mesos master and/or its slave instances.

Getting Started with Mesos-Go

This section of the write up focuses on using the Mesos-Go bindings to create client frameworks in Go.  It provides a high-level overview of the API and and the necessary code examples that shows you how to write a simple Mesos Scheduler.


Building the Code

From your command prompt do the following steps:

$ cd $GOPATH/src/$ mkdir -p github.com/mesos/mesos-go$ git clone https://github.com/mesos/mesos-go.git github.com/mesos/mesos-go$ cd github.com/mesos/mesos-go$ godep restore
$ go install ./...

Developing Mesos Frameworks in Go

Mesos-Go exposes a simple application interface that is aligned with the C++ API to maximize understandability and portability.  In the following sections, we explore how to use the API to create a simple example Mesos framework found in the examples/ directory of the source code.  For instructions on how to build the Mesos-Go and the accompanying examples, visit https://github.com/mesos/mesos-go

If you are not familiar with the way Mesos works, now would be a good time to review the Mesos documentation http://mesos.apache.org/documentation/latest/ and its architecture http://mesos.apache.org/documentation/latest/mesos-architecture/.

Mesos-Go Packages

Once you are ready to use the API, you will primarily interact with the following high-level packages:

Writing a Scheduler

Creating a framework scheduler involves two components: a Scheduler and a SchedulerDriver.  All Mesos frameworks must implement the Scheduler interface which, in turn, uses the SchedulerDriver module to handle communication with the Mesos master.  Let us take a closer look at the two components.

The Scheduler Interface

A framework developer must start by implementing the Mesos Scheduler interface in Go.  The Scheduler is where the developer will place code that:

  • decides when to accept or reject resource offers from the Mesos master
  • handles framework disconnection from the master
  • receives status update from other Mesos nodes
  • decides what to do when an executor is lost (see section on Executor later)
  • handles framework errors
  • Etc.

The following shows the full definition of the Scheduler Interface.  The SchedulerDriver (covered next) employs a callback mechanism to delegate Mesos event-handling to the respective scheduler method as listed in the type definition below (link to the Scheduler interface documentation: https://godoc.org/github.com/mesos/mesos-go/scheduler#Scheduler).

type Scheduler interface {
    Registered(SchedulerDriver, *mesos.FrameworkID, *mesos.MasterInfo)
    Reregistered(SchedulerDriver, *mesos.MasterInfo)
    ResourceOffers(SchedulerDriver, []*mesos.Offer)
    OfferRescinded(SchedulerDriver, *mesos.OfferID)
    StatusUpdate(SchedulerDriver, *mesos.TaskStatus)
        *mesos.ExecutorID, *mesos.SlaveID, string)
    SlaveLost(SchedulerDriver, *mesos.SlaveID)
    ExecutorLost(SchedulerDriver, *mesos.ExecutorID, *mesos.SlaveID, int)
    Error(SchedulerDriver, string)

The Scheduler Driver

The SchedulerDriver is a module provided by Mesos-Go that is designed to handle communication between your framework and the Mesos master.  It exposes its services via several methods to:

  • register the framework with the Mesos  master
  • launch tasks to be executed by the framework's executor (covered later)
  • decline incoming resource offers
  • send framework messages to running executors
  • Etc 

To get started with the driver, a framework developer will need to create a new SchedulerDriver instance using the following constructor function found in the scheduler package.

    sched Scheduler,                 // Scheduler instance
    framework *mesos.FrameworkInfo,  // Framework information
    master string,                   // Mesos master address
    credential *mesos.Credential,    // Credential 
) (*MesosSchedulerDriver, error)         // Returns a pointer to driver or error

The constructor function creates an a new instance of the SchedulerDriver or return an error if something went wrong.  Notice that you must provide an instance of the your framework's scheduler (covered earlier) as a parameter to the the function call.  This is how to specify which scheduler implementation to use for the framework.  The code sample in the next section shows the workflow of creating a fully working sample framework.

Link to SchedulerDriver documetnation - https://godoc.org/github.com/mesos/mesos-go/scheduler#SchedulerDriver.

A Sample Scheduler Framework

The following code snippet is from examples/test_framework.go (https://github.com/mesos/mesos-go/blob/master/examples/test_framework.go. The abbreviated code below It shows the implementation of a simple Mesos framework (but see the original for full details) (abbreviated). The first portion of the code implements the Scheduler interface (again, not all implemented methods are shown).

type ExampleScheduler struct {
executor      *mesos.ExecutorInfo
tasksLaunched int
tasksFinished int
totalTasks    int

func (sched *ExampleScheduler) Registered(
driver sched.SchedulerDriver, 
frameworkId *mesos.FrameworkID, 
masterInfo *mesos.MasterInfo) {...}
func (sched *ExampleScheduler) ResourceOffers(
driver sched.SchedulerDriver, offers []*mesos.Offer) { ... }
func (sched *ExampleScheduler) Error(
driver sched.SchedulerDriver, err string){...}

The remainder of the code sample shows the creation of a new SchedulerDriver instance and the launch of the framework:

func main() {
// the command to be executed by slave processes
exec := &mesos.ExecutorInfo{
ExecutorId: util.NewExecutorID("default"),
Name:       proto.String("Test Executor (Go)"),
Source:     proto.String("go_test"),
Command:    util.NewCommandInfo("/tmp/test-executor"),

// the framework info
fwinfo := &mesos.FrameworkInfo{
User: proto.String(""), // Mesos-go will fill in user.
Name: proto.String("Test Framework (Go)"),

//create the SchedulerDriver instance
     driver, err := scheduler.NewMesosSchedulerDriver(
if stat, err := driver.Run(); err != nil {
    log.Infof("Framework stopped... %s and error: %s\n",
    stat.String(), err.Error())

The variables exec and fwinfo store Protocol buffer objects needed to describe the executor that will be launched and describe the framework that will be registered respectively.  Once the SchedulerDriver is created, it is used to start and register the framework with Mesos master using driver.Run() method which will block until driver.Stop() is called.

Writing an Executor

The second half of writing a Mesos framework is to implement an executor that specify the distributed task to be done by the slave nodes in the cluster.  Similar to the scheduler, creating a framework executor requires two components: an Executor interface and an ExecutorDriver.  The executor is the piece of code that actually runs on the slave nodes, in a Mesos cluster, to performs useful work.  The Executor, in turn, uses the ExecutorDriver  to handle communication with the Mesos slaves.  Let us take a closer look at these components.

The Executor Interface

The Executor interface helps defines the unit of work that will be done by a slave node in the distributed cluster.  When the executor driver receives framework events, it delegates handling of these events to an instance of the Executor implemented by the programmer.  Each method, on the Executor interface, matches a specific framework event to:

  • launch and run task (or tasks) requested by the framework scheduler
  • handle disconnection from the running slave
  • receive messages from the framework
  • handle request to kill the running task
  • Etc.

The following shows the full definition of the Executor interface (link to documentation - https://godoc.org/github.com/mesos/mesos-go/executor#Executor).  

type Executor interface {
Registered(ExecutorDriver, *mesosproto.ExecutorInfo, 
*mesosproto.FrameworkInfo, *mesosproto.SlaveInfo)
Reregistered(ExecutorDriver, *mesosproto.SlaveInfo)
LaunchTask(ExecutorDriver, *mesosproto.TaskInfo)
KillTask(ExecutorDriver, *mesosproto.TaskID)
FrameworkMessage(ExecutorDriver, string)
Error(ExecutorDriver, string)

The main method of the Executor is its LaunchTask().  It is invoked when an associated Scheduler requests a task to be launched on the cluster.  The implementation of the task depends on the the specification of the framework.  It can simply launch an OS process, start new threads, or do a simple computation.

The Executor Driver

The ExecutorDriver is implemented by Mesos-Go and handles interactions between the framework executor and the slave node executing the task.  The driver several methods including the ability to send status update to an associated scheduler and to communicate with the scheduler by sending arbitrary messages.  To create the executor driver, use the following constructor function found in the executor package.

executor.NewMesosExecutorDriver(exec Executor) (*MesosExecutorDriver, error)

The driver constructor function will return a new instance of MesosExecutorDriver or an error if something went wrong.  Notice that you must provide an instance of the your framework's executor as a parameter to the the function call.  This allows you to specify an Executor implementation that will execute a given task whenever the driver receives a request to launch task.  The code sample in the next section shows the workflow for creating a fully working sample executor.

Sample Executor

The following code snippet -- from examples/test_executor.go (https://github.com/mesos/mesos-go/blob/master/examples/test_executor.go) -- shows an abbreviated example of an executor.  The first portion of the code shows an implementation of the Executor interface.

// Type for implementing Executor
type exampleExecutor struct {
tasksLaunched int
func (exec *exampleExecutor) Registered(
driver exec.ExecutorDriver, 
execInfo *mesos.ExecutorInfo,
fwinfo *mesos.FrameworkInfo, 
slaveInfo *mesos.SlaveInfo) {...}

func (exec *exampleExecutor) LaunchTask(
driver exec.ExecutorDriver, taskInfo *mesos.TaskInfo) {...}
func (exec *exampleExecutor) Error(
driver exec.ExecutorDriver, err string) {...}

The remainder of the the code uses the executor driver to launch the executor and register it with the slave process.

func main(){
driver, err := exec.NewMesosExecutorDriver(newExampleExecutor())
if err != nil {
fmt.Println("Unable to create a ExecutorDriver ", err.Error())
if stat, err := driver.Run(); err != nil {
    log.Infof("Executor stopped %s and error: %s\n", 
    stat.String(), err.Error())

The variable driver is used to start the executor and wait for events from the slave process.

Give it a Go!

As one of the newest members of the growing number of the pure Mesos language bindings, Mesos-Go brings a complete set of functionalities that allows developers to create Mesos frameworks using the popular language Go.  Having a native binding means programmers can create Mesos artifacts that can be deployed as single executables.  We invite you to give it a try and see what you can build with it.


java ,cloud ,devops ,frameworks ,deployment ,go ,mesos ,apache mesos

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}