TensorFlow in Kubernetes in 88 MB

TensorFlow is a beast. It deals with machine learning algorithms, it uses Bazel to be built, it uses gRPC, etc., but let’s be honest, you are dying to play with machine learning. Come on, you know you want to! Especially in combination with Docker and Kubernetes. At Bitnami, we love apps, so we wanted to!

TensorFlow + Kubernetes + Docker + Machine learning = Awesomeness.

You just need to add bi-modal in there and you will hit buzzword bingo.

Jokes aside, TensorFlow is an open source library for machine learning. You can train data models and use those models to predict or infer information. I did not dig into TensorFlow itself, but I hear it is using some advanced neural networks techniques. NN have been around for a while, but due to computational complexity, they were not extremely useful passed a couple of layers and a couple dozen neurons (20 years ago at least :) ). Once you have trained a network with some data, you can use that network (aka model) to predict results for data not used in the training data set. As a side note, I am pretty sure we will soon see a marketplace of TF models.

The Inception model is a TensorFlow model trained using the ImageNet library. With that model, you give it an image and TensorFlow (TF) tells you what that image is. In March last year, there was a blog about how to run the inception model in Kubernetes. Great! Except that with things being what they are… what is described in the blog is now broken!

So I am going to share a few tidbits about trying out your first TensorFlow model (the Inception model) and running it in Kubernetes using an 88 MB Docker image. If you are courageous enough to figure out what is broken with the original blog post, you will end up with a ~4.2 GB image, and I think it is way too big and unpractical.

We are working on this at Bitnami to develop a production Helm Chart for TensorFlow and use it with our serverless solution Kubeless.

You can find our (Tomas Pizzaro @tompizmor and I ) WIP right here.

The Build

We followed the general build instructions but replaced the base image with our own minideb, a minimalist Debian Docker image. We also put the Bazel build instructions in the Dockerfile, which allows us to build TensorFlow with a single Docker build. It works great if you give enough RAM to Docker and limit the local resources used by Bazel.

build -t tensorflow -f Dockerfile.minideb .

You can run this image, but it is BIG (~4.2 GB), and it does not serve any actual trained data model, so it is useless in itself. This is just running the TF serving server.

The build process is less than ideal and we will integrate it with our internal automated toolchain, but for now it will do.

To make this easier and because I refuse to pull a 4.2 GB image every time I want to run this, we did a super terrible hacky thing. We just did an ldd on the TensorFlow serving binary, copied the binary and libraries out of the image, copied them into a minimal container based on minideb, and got an 88MB image:

docker pull bitnami/tf-server-slim:0.0.1
docker images | grep tf-server
bitnami/tf-server-slim   0.0.1  1f09190e5b3e 2 hours ago   88.6 MB

Loading the Inception Model in Kubernetes

To load the Inception model, we decided to use a Kubernetes PVC and a Kubernetes Job. That way, we seed the PVC with the model. The Job is just curling the Inception model and sticking it into the PVC. You can try all of this on minikube.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: seed
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 500Mi
---
apiVersion: batch/v1
kind: Job
metadata:
  name: seed-inception
spec:
  template:
    metadata:
      name: seed-inception
    spec:
      containers:
      - name: seed
        image: bitnami/minideb-extras:jessie
        command: ["/bin/sh"]
        args: ["-c", "curl -o /seed/inception-v3-2016-03-01.tar.gz http://download.tensorflow.org/models/image/imagenet/inception-v3-2016-03-01.tar.gz && cd /seed/ && tar -xzf inception-v3-2016-03-01.tar.gz && rm inception-v3-2016-03-01.tar.gz"]
        volumeMounts:
        - name: seed
          mountPath: /seed
      restartPolicy: Never
      volumes:
      - name: seed
        persistentVolumeClaim:
          claimName: seed

Starting the TF server

Now that we have the Inception model data in a PVC, we can run the TF serving server as a deployment and use an init-container to load the model before serving it.

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: tensorflow-serving
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: tf
    spec:
      initContainers:
      - name: load-model
        image: bitnami/tf-server-slim:0.0.1
        command: ['/serving/bazel-bin/tensorflow_serving/example/inception_saved_model', '--checkpoint_dir=/seed/inception-v3', '--output_dir=/serving/inception-export']
        volumeMounts:
        - name: model
          mountPath: "/serving/inception-export"
        - name: seed
          mountPath: "/seed"
      containers:
      - name: serving
        image: bitnami/tf-server-slim:0.0.1
        volumeMounts:
        - name: model
          mountPath: "/serving/inception-export"
      volumes:
      - name: seed
        persistentVolumeClaim:
          claimName: seed
      - name: model
        emptyDir: {}

Using the TF Inception Model

Now that the Inception model is being served, you can expose the deployment via a Kubernetes service and run the client to use it.

kubectl expose deployment tensorflow --port=9000 --type=NodePort

And locally on your machine, run the client container (actually bigger than the server because we have not yet slimmed down the Python client fully) and stream an image to the server (note that the streaming uses Python grpc).

$ docker run --rm -it -v ~/Downloads/:/tmp \
                      --net=host \
                      --entrypoint=/bin/bash \
                      bitnami/tf-client-slim:0.0.1
# bazel-bin/tensorflow_serving/example/inception_client \
            --server=192.168.99.100:32006 \
            --image=/tmp/labrador.jpg

Give it a beautiful image like this:

And your super buzzword friendly setup of minikube Kubernetes with the TensorFlow Inception model will spit out:

outputs {
    key: "classes"
    value {
        dtype: DT_STRING tensor_shape {
            dim {
                size: 1
            }
            dim {
                size: 5
            }
        }
        string_val: "Labrador retriever"
        string_val: "golden retriever"
        string_val: "Rottweiler"
        string_val: "Rhodesian ridgeback"
        string_val: "Chesapeake Bay retriever"
    }
}
outputs {
    key: "scores"
    value {
        dtype: DT_FLOAT tensor_shape {
            dim {
                size: 1
            }
            dim {
                size: 5
            }
        }
        float_val: 8.12877559662 float_val: 6.05589342117 float_val: 4.21776771545 float_val: 3.91829943657 float_val: 3.65974068642
    }
}

Which I believe tells you that it is 81% certain that this is a Labrador retriever and also tells you that I need to dig deeper into the results of the inception model. :) 

If you want to see more blogs like this, don't forget to reach out on Twitter @sebgoa.