ZooKeeper on Kubernetes

I've never tried that myself, I can't tell if its a good or bad practice, I can see some benefits, but I can also see that this is something I am not really excited about. It could look like this:

docker run -p 2181:2181 -v /path/to/my/conf:/opt/zookeeper/conf my/zookeeper

server.1=server1.example.com:2888:3888
server.2=server2.example.com:2888:3888
server.3=server3.example.com:2888:3888
...
server.current=[bind address]:[peer binding port]:[election biding port]Note that if the server id is X the server.X entry needs to contain the bind ip and ports and not the connection ip and ports.



So what we actually need to pass to the container as environment variables are the following:

if [ ! -z "$SERVER_ID" ]; then
  echo "$SERVER_ID" > /opt/zookeeper/data/myid
  #Find the servers exposed in env.
  for i in `echo {1..15}`;do

    HOST=`envValue ZK_PEER_${i}_SERVICE_HOST`
    PEER=`envValue ZK_PEER_${i}_SERVICE_PORT`
    ELECTION=`envValue ZK_ELECTION_${i}_SERVICE_PORT`

    if [ "$SERVER_ID" = "$i" ];then
      echo "server.$i=0.0.0.0:2888:3888" >> conf/zoo.cfg
    elif [ -z "$HOST" ] || [ -z "$PEER" ] || [ -z "$ELECTION" ] ; then
      #if a server is not fully defined stop the loop here.
      break
    else
      echo "server.$i=$HOST:$PEER:$ELECTION" >> conf/zoo.cfg
    fi

  done
fi

ZooKeeper on Kubernetes

{
                                    "image": "fabric8/zookeeper",
                                    "name": "zookeeper-server-1",
                                    "env": [
                                        {
                                            "name": "ZK_SERVER_ID",
                                            "value": "1"
                                        }
                                    ],
                                    "ports": [
                                        {
                                            "name": "zookeeper-client-port",
                                            "containerPort": 2181,
                                            "protocol": "TCP"
                                        },
                                        {
                                            "name": "zookeeper-peer-port",
                                            "containerPort": 2888,
                                            "protocol": "TCP"
                                        },
                                        {
                                            "name": "zookeeper-election-port",
                                            "containerPort": 3888,
                                            "protocol": "TCP"
                                        }
                                    ]
                                }

• ZK_PEER_1_SERVICE_HOST
• ZK_PEER_1_SERVICE_PORT
• ZK_ELECTION_1_SERVICE_PORT
• ZK_PEER_2_SERVICE_HOST
• ZK_PEER_2_SERVICE_PORT
• ZK_ELECTION_2_SERVICE_PORT
• ZK_PEER_3_SERVICE_HOST
• ZK_PEER_3_SERVICE_PORT
• ZK_ELECTION_3_SERVICE_PORT

• the name of the port
• the kubernetes pod the provide the service

"labels": {
    "name": "zookeeper-pod",
    "server":  1
}

Something like the above could work. Now we are ready to define the service. I will just show how we can expose the peer port of server with id 1, as a service. The rest can be done in a similar fashion:

{
    "apiVersion": "v1beta1",
    "creationTimestamp": null,
    "id": "zk-peer-1",
    "kind": "Service",
    "port": 2888,
    "containerPort": "zookeeper-peer-port",
    "selector": {
        "name": "zookeeper-pod",
        "server": 1
    }
}

The basic idea is that in the service definition, you create a selector which can be used to query/filter pods. Then you define the name of the port to expose and this is pretty much it. Just to clarify, we need a service definition just like the one above per zookeeper server container. And of course we need to do the same for the election port.

Finally, we can define an other kind of service, for the client connection port. This time we are not going to specify the sever id, in the selector, which means that all 3 servers will be selected. In this case kubernetes will load balance across all ZooKeeper servers. Since ZooKeeper provides a single system image (it doesn't matter on which server you are connected) then this is pretty handy.

{
    "apiVersion": "v1beta1",
    "creationTimestamp": null,
    "id": "zk-client",
    "kind": "Service",
    "port": 2181,
    "createExternalLoadBalancer": "true",
    "containerPort": "zookeeper-client-port",
    "selector": {
        "name": "zookeeper-pod"
    }
}

Finally, we can define an other kind of service, for the client connection port. This time we are not going to specify the sever id, in the selector, which means that all 3 servers will be selected. In this case kubernetes will load balance across all ZooKeeper servers. Since ZooKeeper provides a single system image (it doesn't matter on which server you are connected) then this is pretty handy.

{
    "apiVersion": "v1beta1",
    "creationTimestamp": null,
    "id": "zk-client",
    "kind": "Service",
    "port": 2181,
    "createExternalLoadBalancer": "true",
    "containerPort": "zookeeper-client-port",
    "selector": {
        "name": "zookeeper-pod"
    }
}