Running the Nanos Unikernel Inside Firecracker

A lot of people seem to be under the impression that Firecracker is a competing technology against unikernels. It actually isn't. It's more of an alternative to existing machine monitors such as qemu and is actually complementary if you need the quick boot times. As for isolation levels you are going to have about the same isolation as you would get with KVM - it uses the same facilities underneath.

OSv was the first unikernel to have support for Firecracker but now Nanos has support as well.

There is another very common misconception that firecracker somehow makes your application go faster at runtime. This is, at least today, definitely not the case as discussed in this issue. If anything you will notice your application slows down considerably. What is faster is the boot time. The boot time, if you are concerned about such a thing, does indeed become much faster, although again, take this with a few caveats. If you are sticking something like the JVM or rails inside firecracker your boot time arguments get effectively thrown away because they are already incredibly slow to boot. I suppose comparing with a linux instance you would shave off some but comparing with a unikernel not so much. This isn't necessarily a bad thing.

As indicated in some of the linked mailing list comments there seem to be plans on fixing the runtime performance, however, you should be aware of what you are getting into.

Ok, enough disclaimers - let's get with the code.

Install Firecracker and Nanos

Before you try anything out be aware that none of this will work without having access to hardware acceleration. So if you are in the cloud that means nested virtualization on GCloud or metal instances on AWS otherwise stick to bare metal.

A quick way to check is to grep your cpuinfo:

xxxxxxxxxx

grep -woE 'svm|vmx' /proc/cpuinfo | uniq

There's a handful of other cpu flags that aren't documented that you'll need but suffice to say just make sure your server was built in the past 5-6 years and you'll probably be good. You might be asking how OPS and Nanos can deploy to the cloud without relying on special instance types. The reason is OPS deploys raw unikernels straight to the cloud. Not having an underlying linux means you get access to the raw virtualization inherently present, not to mention you won't have to deal with orchestration.

Ok, first things first, let's install firecracker:

xxxxxxxxxx

latest=$(basename $(curl -fsSLI -o /dev/null -w  %{url_effective} https://github.com/firecracker-microvm/firecracker/releases/latest))

curl -LOJ https://github.com/firecracker-microvm/firecracker/releases/download/${latest}/firecracker-${latest}-$(uname -m)

mv firecracker-${latest}-$(uname -m) firecracker

chmod +x firecracker

Now let's install ops:

xxxxxxxxxx

curl https://ops.city/get.sh -sSfL | sh

For the purposes of this article we'll only be using ops to build our unikernel - not run it. We'll leave the running to firecracker.

If this is your first time using unikernels or OPS please take the time to go through some hello world tutorials such as https://medium.com/javascript-in-plain-english/build-your-first-unikernel-with-express-js-and-nanovms-21d64ff7231c or https://codeforgeek.com/build-and-run-your-first-javascript-unikernel/. They go more in depth on getting your first hello world off the ground and then you can come back to this article which is a bit more advanced.

If you wish to run the unikernel in firecracker you can use this vm_config.json template below.

As you will note I've pointed the kernel_image_path to the stage3 from Nanos found in ~/.ops/0.1.26/stage3.img and I've pointed the image to whatever image you've built with ops. In this case it is located in ~/.ops/images/my_img.img. Other than that everything is fairly vanilla.

{

  "boot-source": {

    "kernel_image_path": "/Users/bob/.ops/0.1.26/stage3.img",

    "boot_args": "console=ttyS0 reboot=k panic=1 pci=off"

  },

  "drives": [

    {

      "drive_id": "rootfs",

      "path_on_host": "/Users/bob/.ops/images/my_img.img",

      "is_root_device": true,

      "is_read_only": false

    }

  ],

  "network-interfaces": [

    {

      "iface_id": "eth0",

      "guest_mac": "AA:FC:00:00:00:01",

      "host_dev_name": "tap0"

    }

  ],

  "machine-config": {

    "vcpu_count": 1,

    "mem_size_mib": 1024,

    "ht_enabled": false

  }

}

Next you'll want to install a DHCP server - cause without it you really can't do anything. Keep in mind the only way to talk to a unikernel is over the network and there is no way to login to these. Also, using firecracker kind of implies that you'll probably have multi-tenants to begin with.

xxxxxxxxxx

sudo apt-get install isc-dhcp-server

You can setup the dhcp server with this config like so:

(Found in /etc/dhcp/dhcpd.conf):

xxxxxxxxxx

option domain-name "example.org";

option domain-name-servers ns1.example.org, ns2.example.org;

default-lease-time 600;

max-lease-time 7200;

ddns-update-style none;

INTERFACES="tap0";

subnet 10.0.2.0 netmask 255.255.255.0 {

    option routers 10.0.2.1;

    range 10.0.2.10 10.0.2.255;

}

Create a tap device for your dhcp server to listen in on:

xxxxxxxxxx

sudo ip tuntap add dev tap0 mode tap

sudo ip addr add 10.0.2.1/24 dev tap0

sudo ip link set tap0 up

Run it against your tap device. You'll want this running before running firecracker.

xxxxxxxxxx

dhcpd -f -d tap0

You should see some ARP requests fly by once firecracker boots:

x

bob@box:/home/bob~ dhcpd -f -d tap0

Internet Systems Consortium DHCP Server 4.3.5

Copyright 2004-2016 Internet Systems Consortium.

All rights reserved.

For info, please visit https://www.isc.org/software/dhcp/

Config file: /etc/dhcp/dhcpd.conf

Database file: /var/lib/dhcp/dhcpd.leases

PID file: /var/run/dhcpd.pid

lease 10.0.2.0: no subnet.

Wrote 0 leases to leases file.

Listening on LPF/tap0/96:ea:ca:e0:76:63/10.0.2.0/24

Sending on   LPF/tap0/96:ea:ca:e0:76:63/10.0.2.0/24

Sending on   Socket/fallback/fallback-net

Server starting service.

DHCPDISCOVER from aa:fc:00:00:00:01 via tap0

DHCPOFFER on 10.0.2.10 to aa:fc:00:00:00:01 via tap0

DHCPREQUEST for 10.0.2.10 (10.0.2.1) from aa:fc:00:00:00:01 via tap0

DHCPACK on 10.0.2.10 to aa:fc:00:00:00:01 (uniboot) via tap0

DHCPREQUEST for 10.0.2.10 from aa:fc:00:00:00:01 (uniboot) via tap0

DHCPACK on 10.0.2.10 to aa:fc:00:00:00:01 (uniboot) via tap0

DHCPREQUEST for 10.0.2.10 from aa:fc:00:00:00:01 (uniboot) via tap0

DHCPACK on 10.0.2.10 to aa:fc:00:00:00:01 (uniboot) via tap0

DHCPREQUEST for 10.0.2.10 from aa:fc:00:00:00:01 (uniboot) via tap0

DHCPACK on 10.0.2.10 to aa:fc:00:00:00:01 (uniboot) via tap0

Once you boot your unikernel you should also see it grab an ip:

xxxxxxxxxx

Server started on port 8080

assigned: 10.0.2.10

assigned: 0.0.0.0

I've found it easier to play with firecracker by breaking up the one large default vm_config.json into multiple commands each with different configs. I've also set up logging cause frankly it's a pain to understand what is going on without it.

Boot.sh:

This sets up the entry point for firecracker. Each socket needs to be unique so once you are done with it don't forget to remove it if you wish to re-use it later on. For this initial boot we simply point it at stage3 found in the latest ops release we have:

xxxxxxxxxx

#!/bin/sh

curl --unix-socket /tmp/firecracker.socket -i \

-X PUT 'http://localhost/boot-source' \

-H 'Accept: application/json' \

-H 'Content-Type: application/json' \

-d '{

"kernel_image_path": "/home/bob/.ops/0.1.26/stage3.img",

"boot_args": "console=ttyS0 reboot=k panic=1 pci=off" }'

Drives.sh:

This one points firecracker at our root fs that gets loaded. This is the actual image that you are building with ops. We treat this differently than ops does as we bypass a bunch of booting setup that we'd normally have to do using something like KVM.

xxxxxxxxxx

#!/bin/sh

curl --unix-socket /tmp/firecracker.socket -i \

-X PUT 'http://localhost/drives/rootfs' \

-H 'Accept: application/json' \

-H 'Content-Type: application/json' \

-d '{

      "drive_id": "rootfs",

      "path_on_host": "/home/bob/.ops/0.1.26/images/my_img.img",

      "is_root_device": true,

      "is_read_only": false

}'

}'

Machine.sh:

This sets the machine parameters for our firecracker instance. Nothing special here.

xxxxxxxxxx

#!/bin/sh

 curl --unix-socket /tmp/firecracker.socket -i \

-X PUT 'http://localhost/machine-config' \

-H 'Accept: application/json' \

-H 'Content-Type: application/json' \

-d '{

    "vcpu_count": 1,

    "mem_size_mib": 1024,

    "ht_enabled": false

}'

Start.sh:

Next we start the instance:

xxxxxxxxxx

#!/bin/sh

curl --unix-socket /tmp/firecracker.socket -i \

-X PUT 'http://localhost/actions' \

-H 'Accept: application/json' \

-H 'Content-Type: application/json' \

-d '{

"action_type": "InstanceStart"

}'

Logs.sh:

Then we'll create 2 pipes and attach the logs to them. Note: You most definitely should do this. I'm not quite sure why this is so obtuse or why the documentation for this was hidden inside of a github ticket. The logs are invaluable to understand what is going on in firecracker when things "just don't work".

xxxxxxxxxx

#!/bin/sh

mkfifo log.fifo

mkfifo metrics.fifo

curl --unix-socket /tmp/firecracker.socket -i \

-X PUT 'http://localhost/logger' \

-H "accept: application/json" \

-H "Content-Type: application/json" \

-d '{ "log_fifo": "log.fifo", "metrics_fifo": "metrics.fifo", "level":

"Info", "show_level": true, "show_log_origin": true }'

Read_fifo.sh:

To read your logs you'll need this little script:

xxxxxxxxxx

#!/bin/bash

while true

do

    if read line <$1; then

        if [[ "$line" == 'quit' ]]; then

            break

        fi

        echo $line

    fi

done

echo "Reader exiting"

Then to actually read the logs:

xxxxxxxxxx

./read_fifo.sh log.fifo