TensorFlow in Kubernetes in 88 MB
Machine learning, containers, and serverless, oh my! You can pack a TensorFlow model into Docker to save space, then orchestrate it with Kubernetes.
Join the DZone community and get the full member experience.
Join For FreeTensorFlow is a beast. It deals with machine learning algorithms, it uses Bazel to be built, it uses gRPC, etc., but let’s be honest, you are dying to play with machine learning. Come on, you know you want to! Especially in combination with Docker and Kubernetes. At Bitnami, we love apps, so we wanted to!
TensorFlow + Kubernetes + Docker + Machine learning = Awesomeness.
You just need to add bi-modal in there and you will hit buzzword bingo.
Jokes aside, TensorFlow is an open source library for machine learning. You can train data models and use those models to predict or infer information. I did not dig into TensorFlow itself, but I hear it is using some advanced neural networks techniques. NN have been around for a while, but due to computational complexity, they were not extremely useful passed a couple of layers and a couple dozen neurons (20 years ago at least :) ). Once you have trained a network with some data, you can use that network (aka model) to predict results for data not used in the training data set. As a side note, I am pretty sure we will soon see a marketplace of TF models.
The Inception model is a TensorFlow model trained using the ImageNet library. With that model, you give it an image and TensorFlow (TF) tells you what that image is. In March last year, there was a blog about how to run the inception model in Kubernetes. Great! Except that with things being what they are… what is described in the blog is now broken!
So I am going to share a few tidbits about trying out your first TensorFlow model (the Inception model) and running it in Kubernetes using an 88 MB Docker image. If you are courageous enough to figure out what is broken with the original blog post, you will end up with a ~4.2 GB image, and I think it is way too big and unpractical.
We are working on this at Bitnami to develop a production Helm Chart for TensorFlow and use it with our serverless solution Kubeless.
You can find our (Tomas Pizzaro @tompizmor and I ) WIP right here.
The Build
We followed the general build instructions but replaced the base image with our own minideb, a minimalist Debian Docker image. We also put the Bazel build instructions in the Dockerfile, which allows us to build TensorFlow with a single Docker build. It works great if you give enough RAM to Docker and limit the local resources used by Bazel.
build -t tensorflow -f Dockerfile.minideb .
You can run this image, but it is BIG (~4.2 GB), and it does not serve any actual trained data model, so it is useless in itself. This is just running the TF serving server.
The build process is less than ideal and we will integrate it with our internal automated toolchain, but for now it will do.
To make this easier and because I refuse to pull a 4.2 GB image every time I want to run this, we did a super terrible hacky thing. We just did an ldd on the TensorFlow serving binary, copied the binary and libraries out of the image, copied them into a minimal container based on minideb, and got an 88MB image:
docker pull bitnami/tf-server-slim:0.0.1
docker images | grep tf-server
bitnami/tf-server-slim 0.0.1 1f09190e5b3e 2 hours ago 88.6 MB
Loading the Inception Model in Kubernetes
To load the Inception model, we decided to use a Kubernetes PVC and a Kubernetes Job. That way, we seed the PVC with the model. The Job is just curling the Inception model and sticking it into the PVC. You can try all of this on minikube.
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: seed
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 500Mi
---
apiVersion: batch/v1
kind: Job
metadata:
name: seed-inception
spec:
template:
metadata:
name: seed-inception
spec:
containers:
- name: seed
image: bitnami/minideb-extras:jessie
command: ["/bin/sh"]
args: ["-c", "curl -o /seed/inception-v3-2016-03-01.tar.gz http://download.tensorflow.org/models/image/imagenet/inception-v3-2016-03-01.tar.gz && cd /seed/ && tar -xzf inception-v3-2016-03-01.tar.gz && rm inception-v3-2016-03-01.tar.gz"]
volumeMounts:
- name: seed
mountPath: /seed
restartPolicy: Never
volumes:
- name: seed
persistentVolumeClaim:
claimName: seed
Starting the TF server
Now that we have the Inception model data in a PVC, we can run the TF serving server as a deployment and use an init-container to load the model before serving it.
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: tensorflow-serving
spec:
replicas: 1
template:
metadata:
labels:
app: tf
spec:
initContainers:
- name: load-model
image: bitnami/tf-server-slim:0.0.1
command: ['/serving/bazel-bin/tensorflow_serving/example/inception_saved_model', '--checkpoint_dir=/seed/inception-v3', '--output_dir=/serving/inception-export']
volumeMounts:
- name: model
mountPath: "/serving/inception-export"
- name: seed
mountPath: "/seed"
containers:
- name: serving
image: bitnami/tf-server-slim:0.0.1
volumeMounts:
- name: model
mountPath: "/serving/inception-export"
volumes:
- name: seed
persistentVolumeClaim:
claimName: seed
- name: model
emptyDir: {}
Using the TF Inception Model
Now that the Inception model is being served, you can expose the deployment via a Kubernetes service and run the client to use it.
kubectl expose deployment tensorflow --port=9000 --type=NodePort
And locally on your machine, run the client container (actually bigger than the server because we have not yet slimmed down the Python client fully) and stream an image to the server (note that the streaming uses Python grpc).
$ docker run --rm -it -v ~/Downloads/:/tmp \
--net=host \
--entrypoint=/bin/bash \
bitnami/tf-client-slim:0.0.1
# bazel-bin/tensorflow_serving/example/inception_client \
--server=192.168.99.100:32006 \
--image=/tmp/labrador.jpg
Give it a beautiful image like this:

And your super buzzword friendly setup of minikube Kubernetes with the TensorFlow Inception model will spit out:
outputs {
key: "classes"
value {
dtype: DT_STRING tensor_shape {
dim {
size: 1
}
dim {
size: 5
}
}
string_val: "Labrador retriever"
string_val: "golden retriever"
string_val: "Rottweiler"
string_val: "Rhodesian ridgeback"
string_val: "Chesapeake Bay retriever"
}
}
outputs {
key: "scores"
value {
dtype: DT_FLOAT tensor_shape {
dim {
size: 1
}
dim {
size: 5
}
}
float_val: 8.12877559662 float_val: 6.05589342117 float_val: 4.21776771545 float_val: 3.91829943657 float_val: 3.65974068642
}
}
Which I believe tells you that it is 81% certain that this is a Labrador retriever and also tells you that I need to dig deeper into the results of the inception model. :)
If you want to see more blogs like this, don't forget to reach out on Twitter @sebgoa.
Published at DZone with permission of Sebastien Goasguen, DZone MVB. See the original article here.
Opinions expressed by DZone contributors are their own.
Comments