Over a million developers have joined DZone.

The Cloudcast #155 - Large Scale Distributed Infrastructure with Mesos

DZone's Guide to

The Cloudcast #155 - Large Scale Distributed Infrastructure with Mesos

· Cloud Zone ·
Free Resource

Discover a centralized approach to monitor your virtual infrastructure, on-premise IT environment, and cloud infrastructure – all on a single platform.

Download the MP3

Date: July 31, 2014

By: Aaron Delp and Brian Gracely

Description: Aaron and Brian talk with Dave Lester (@davelester, Open Source Advocate at Twitter. Friend of @ApacheMesos, @ApacheAurora) about how Twitter manages large-scale infrastructure, an introduction to Apache Mesos and how projects like Kubernetes, Docker, Aurora are helping to define the next-generation of web-scale infrastructure management. Music Credit: Nine Inch Nails (www.nin.com)

Today’s show is dedicated to the memory of a good friend of the show, Jim Ruddy, who left us last week in a tragic car accident. Jim loved his family, life and exploring all the new technology that we cover on the show. He’ll be deeply missed by everybody that knew him.

Topic 1 - We met at OSCON last week and you introduced me to Apache Mesos. Give us some of your background and what you do at Twitter with regard to Mesos?

Topic 2 - Let’s talk about the types of challenges that Mesos attempts to solve at Twitter and why so many webscale companies are using it. (Twitter, AirBnB, CloudPhysics, eBay, Netflix, PayPal, Vimeo, etc.)

Topic 3 - As I started digging into the Mesos architecture, it’s a fairly complex set of inter-related activities (Mesos Controller, Hadoop, ZooKeeper, Docker). Can you walk us through the basics of what area function does or some common use-cases?
  • Zookeeper - like a CMDB - keeping track of configs
  • Hadoop Scheduler - collecting data to feed the Mesos scheduler?
  • Mesos Master - making scheduling decisions (using hadoop data) and pulling configs from Zookeeper?

Topic 4 - Mesos appears to be primarily focused on compute, but does it have any logic to also interact with networking elements (network segments, load-balancers, firewalls, etc.)?

Topic 5 - Is Mesos a system that could live on top of something like OpenStack, or is it primarily focused on containers or bare-metal compute clusters?

Topic 5 - Some of our listeners have heard about this thing that Google released, called Kubernetes, which they use to build their Google Compute Engine. On some level it sounds like another resource scheduler, but we also heard about Mesos + Kubernetes interaction. Are these groups solving similar problems, or are they finding new ways to use tools/frameworks that might have been used in other ways before (eg. Hadoop)?

Topic 6 - There is an event happening in August (21st-22nd) called MesosCon, in Chicago (along with LinuxCon). What are the interesting things that will be happening those days?

Learn how to auto-discover your containers and monitor their performance, capture Docker host and container metrics to allocate host resources, and provision containers.


Published at DZone with permission of

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}