Vagrant Cloud - Creating Vagrant Box Files for a VirtualBox Provider in AWS

[This article was written by Michael Sverdlik.]
The Pain Point

Many developers often need to create easily reproducible development environments – for anything from testing to troubleshooting, and even continued development across teams. To this end, many technologies have arisen to answer this need from Vagrant and VirtualBox, and even Docker in certain contexts. However, with the onset of cloud where many companies choose to do dev and test and QA work on resources on demand, this type of virtual development environment comes with its downsides too, namely issues such as nested virtualization.

In the context of our R&D and Ops work on Cloudify, an open source cloud orchestration tool written Python with a TOSCA-based YAML DSL, we often need to create reproducible and portable development environments on the cloud, and had to find a way to overcome these issues. We wanted to find the most seamless process to do this – that too would be easily replicable per environment. So what better way than to start with the most popular cloud – AWS? This post will dive into one such scenario of porting Vagrant .box files for AWS, demonstrating how to overcome issues like nested virtualization and the need for bare metal machines that are also costly and time-consuming to provision. In this post you’ll have a step-by-step tutorial for how to easily create v2v (virtual to virtual) machines, and create a VMDK disk image that can then be uploaded to any AWS environment.

The Demo

For the quick trial of Cloudify, we provide a Vagrantfile and Vagrant box with Cloudify's Manager preinstalled on a VirtualBox image.

By utilizing Vagrant and VirtualBox we are able to provide our customers with a reproducible demo environment to evaluate Cloudify locally on their personal computers.

We could have just provided OVF & VMDK files and have had users import these into VirtualBox; however the point was to make the evaluation as simple as possible, and Vagrant strips away potential issues one might encounter when dealing directly with VirtualBox. So instead of providing a detailed explanation on how to correctly set up a VirtualBox VM, we can summarize our quick start guide in two bullets:

1. Download the Vagrantfile
2. Run vagrant up

Utilizing Packer to create Vagrant boxes

Creating a Vagrant box is a very straightforward matter. You can create one by using Vagrant itself or one of the many utilities available for performing this task.

Packer, one such option, is written by the same guys who wrote Vagrant and is a natural choice for this task. Packer works somewhat similarly to Vagrant; however its focus is on producing images at the end of the process, and not running an environment, as Vagrant does.

For VirtualBox images Packer needs to create a virtual machine on VirtualBox, provision it, and export it at the end of the process as a box file.

Our objective seemed very simple to achieve until the moment we realized that we can't run VirtualBox on our build machines.

The nested Virtualization problem

Here at GigaSpaces, most of our infrastructure is virtualized. While this is great and allows us to better utilize our hardware, it does have some unavoidable "side effects".

One of those side effects is the inability to run VirtualBox inside another VM, since our hypervisor of choice doesn't allow nested virtualization.

Nested virtualization is a feature in virtualization solutions which allows you to run hypervisors inside a hypervisor. Essentially, it allows you to run a VM inside another VM.

Poking around the web, it looks like none of the popular IaaS providers supports nested virtualization either.

Ideas for solutions and new problems

Possible ideas for a solution we had in mind:

1. Use a bare metal box and avoid the problem:

While this is the most obvious solution, it is also the least desirable from our point of view. We don't have any infrastructure in place to support bare metal provisioning. A dedicated bare metal server for a once-a-day image build is a phenomenal waste. Sharing a build machine with other builds is something we're even not going to consider since eventually things break due to collisions between builds, and less than perfect cleanups between one build and the one that comes after.

2. Use specialized solutions that allow nested virtualization:

• Using a hypervisor which supports nested virtualization (VMWare Workstation for example)
• Using solutions from service providers such as Ravello (which piggybacks on AWS)

These solutions provide additional cost (licensing or usage fees) and require us to adapt to unfamiliar APIs which could potentially break our current tool chain (Packer/Vagrant plugins for Ravello anyone?).

3. Create a disk image without starting a VM in VirtualBox:

We can provision a virtual machine, take its disk image and convert it to VMDK. Sounds possible in theory, in practice we had never done this before.

We decided to spend some time on researching alternative number three.

There are several ways to convert physical to virtual (p2v) or virtual to physical (v2p). In our case, what about doing v2v?

AWS allows you to export machines only if they were previously imported by you as well. As an alternative, we can provision a machine, make an image out of its hard drive and convert it to a VMDK image. Then, all that is left is to add an OVF descriptor, bundle everything with Vagrant's metadata and tar it into a .BOX file.

Tools of Craft

• Python as the scripting language with:
• Fabric as the task executor over SSH
• Boto as the API for AWS
• Since we are a Python shop when it comes to Cloudify, this is a natural choice for us.
• Packer - for Cloudify Manager provisioning: Packer is not a must in our case since we can script our way in to replicate what Packer does with Boto and Fabric. We'd rather use Packer since it can replicate the process of creating an image on other providers.
• AWS as the IaaS provider: since we are comfortable with its API, our tools support it, and we can run it virtually everywhere without the need to have access to the office.

The Plan

Here is our step by step plan:

1. Create a source image (AMI) with Cloudify pre-installed by using Packer.
2. Launch a worker instance in AWS with the snapshot or source image as one of its disks.
3. On the worker image - create a raw image volume as a file and create an ext4 partition on it.
4. Copy over the data from the source image disk to the previously created ext4 partition.
5. Install the bootloader (extlinux) on the ext4 partition.
6. Convert the raw image into a VMDK.
7. Bundle the VMDK using an OVF descriptor and Vagrant metadata and create a tar file with the content and .box extension.
8. Upload to S3.
9. Cleanup.

Here's a brief sequence diagram of the planned flow.

>>> Read the full post with the step-by-step walkthrough at Developer.com.