DockerCon Hackathon: Continuous Dockery

last year, team electric cloud participated in the first annual dockercon hackathon, and won as one of the top-three submissions. this year, nikhil and i returned to a bigger and badder hackathon event, evidence of docker’s massive growth.

how it works

40+ teams of 1-10 hackers spent 24 hours working on a project from scratch.  categories for submission:

1. build, ship and run cool apps with docker
2. management & operations: logging, monitoring, ui / kitematic, developer tools, deployment, ci / cd, stats, etc
3. orchestration: composition, scheduling, clustering, service discovery, high availability, load balancing, etc
4. security, compliance, & governance: authorization, provenance, distribution, etc
5. resources: networking, storage api, etc

everyone submitted a 2-minute video, and 10 teams were selected to present.  of those presenting, the judges selected the top 3 as winners.

our plan

electric cloud exists to help people deliver better software faster.  we wanted to show how docker fits in with other tools in the software delivery ecosystem.  being experts in our own software, we decided to use electric cloud products to tie everything together – accelerating end-to-end continuous delivery , using:

• electricflow – an orchestration tool that acts as the single pane of glass from commit through production
• electricaccelerator – an acceleration tool that dramatically speeds up builds and tests by distributing them across a cluster of cpus

last year’s entry focused on the build stage of a continuous delivery pipeline.  this year, we focused on the integration stage.

we built a deployment process that:

1. dynamically spins up a vm on either ec2 or openstack
2. runs docker bench for security tests
3. retrieves artifacts from bintray and docker hub
4. stands up linked mysql and wildfly containers running the application
5. runs selenium tests distributed across a cluster
6. pushes some statistics to a dashing dashboard
7. then automatically tears down the vm if the tests are successful.

the deployment process and the various technologies involved

a lot to accomplish in 24 hours! but we were up for the task – and with less-than-pretty version of this diagram chicken-scratched on a piece of paper, we got to work!

what we built

we chose a sample web application called the heat clinic because it has a couple of moving parts (application server and database) making it a somewhat realistic example.  we started out by building the continuous delivery pipeline.

the continuous delivery pipeline defined in electricflow

for this hackathon, we focused on the integration stage.  still, it’s important to know what the pipeline is – to make sure the automation pieces are reusable , and knowing how they’d be reused is key.  having kept this in mind, everything we built can be plugged in to production (or any other stage) with minimal effort.

the next step was modeling the application .  the heat clinic application has two tiers, one for the web application and one for the database.  each of those tiers has a few different components (artifacts) – the wildfly/mysql containers from docker hub, the war file for the web application, configuration files, sql initialization scripts, etc.  we defined the tiers, the components, and the processes to deploy or undeploy each of those components.

the application model defined in electricflow

next, we defined the deployment process that coordinates everything .  this process is closely aligned with the diagram shown earlier: spin up the dynamic environment, run the security tests, retrieve all the artifacts, stand up the containers (in the right order), run the selenium tests, and tear down the environment if everything is successful.

the deployment process defined in electricflow

the selenium suite we put together took a long time to run, and we realized this is not uncommon for selenium.  so we sped up the selenium test suite by using electricaccelerator .  by distributing the 101 tests across just two 4-core vms, accelerator used its patented secret sauce to parallelize and run the tests on the individual cores, bringing the overall time down from >27 minutes to <4 minutes.  that’s 7 times faster with just 2 machines!  if we were to add more vms to our cluster, we could bring that time down to <30 seconds .  that’s a whopping 60 times faster!

visualizing how electricaccelerator distributed the selenium tests across a cluster

finally, we put a pretty face on our work by pushing some key stats to dashing – typically displayed on a tv screen so everyone has an “at a glance” view of the health of the system.

dashing dashboard showing key statistics

our submission

while we did not win this time around, we did come out with a very cool story and a working set of integrations highlighting docker in the context of continuous delivery .  here are the pain points we looked to address:

• you’re looking at docker but need to tie it together with a bunch of existing tools
• you’re looking to increase your velocity by implementing continuous delivery & continuous testing
• you need to gather and surface critical stats for your applications
• you want to make sure you’re auditing for security at the earliest possible stage
• you want to run your long-running integration tests early and often

check out the entire flow in this short 3 minutes video we included in our submission:

we’re already looking forward to the dockercon hackathon next year.  it will be interesting to see what the rapidly changing docker landscape looks like by then!

how to integrate docker as part of your cd pipeline?

container technology like docker promises to provide version-able, environment-independent application services in a snap. however, the tasks and tools involved in creating, validating, promoting and delivering docker containers into production environments are many, complex and time-consuming.

to learn more on how to successfully incorporate docker as part of your end-to-end continuous delivery pipeline, i invite you to join my colleague nikhil vaze and myself on an upcoming webinar, when we’ll be discussing: