In Docker Adoption Pathway: Part II, we discussed CPU monitoring in containers. We examined choices and different paths for monitoring containers CPU. In addition, we covered the details of the way Docker computes the CPU consumption of a container. We summarized by looking at how the Docker API exposes the way in which we can collect the container’s CPU metrics. We went through that path in order to properly monitor our container as part of our standard data-center monitoring process.
In this post, we are going to continue the monitoring track. However, this time, we will look at how JMX is used for monitoring in containerized environments. JMX is heavily used as a successful Java applications management and monitoring tool since its inception. This means we have to consider its behavior in our new containerized world. By utilizing JMX, we can glance inside our running Java application to collect and analyze behavior. Then, we initiate a JMX connection to a Java application running it. This may sound trivial, but it requires more than a few steps, multiple problem mitigations, to reach a successful outcome. We’ll go through the process.
JMX and Containers
What could possibly be different with containers? You ask a valid question in terms of this and many other transitions of apps to be containerized. Why would exposing and using plain JMX service from a container be any different from exposing and using it on a standard VM process?
These questions are certainly valid and, as we shall see, touch a few sensitive issues with containers. Namely, how do we configure processes in containers? Do we configure them at all or do we ship containers together with their configurations as immutable packages? What is the relationship between orchestrator services and management services such as JMX? How should I allow GUI apps outside the containers cluster to access JMX services that inherently reside inside the cluster? From a deployment perspective, JMX translates to an exposed port. Containers have their own port exposure dynamics. Containers’ orchestrators have services. The typical deployment architecture of a container is an application as a container sitting behind either web apps or plain orchestrator services. Services such as Kubernetes container management system provides round-robin routing for the apps backend, which is not suitable when there is a need to access a specific container JMX.
In this post, we are going to run Java application, Apache Cassandra, and try to connect to its JMX while it’s being run as a container. While most of the changes we are going to perform in order to be able to connect to our Cassandra container do not appear to be directly related to Docker, most of them affect the way we package and run our container so they end up in a close relationship with our containers packaging and deployment.
Let’s Connect to a Container’s JMX
To begin with, we are going to launch a Cassandra instance as a container. Fortunately, using Docker, this can be done using a single command line with correspondence to this Docker image documentation:
“Starting a Cassandra instance is simple: $ docker run --name some-cassandra -d cassandra:tag,” so we are going to run the same command as in documentation only with latest tag:
docker run --name containerized-cassandra -d cassandra 91523e6a1e34f52e89993ae75821633a92b2528c5e0f551983a9518f7044d286
This Cassandra instance serves as our Java application example, which is now up and running as a Docker container.
Let’s connect to its JMX interface. What’s the port, you ask? According to its documentation, it is on 7199 - Cassandra JMX monitoring port.
We are going to connect to the JMX like a pro, which means not with UI but with a command line! For this, someone already prepared a nice utility (jmxterm jmx command line client). We are going to ask it to connect to the Cassandra JMX using this command:
$ java -jar ~/Downloads/jmxterm-1.0-alpha-4-uber.jar --url localhost:7199
Alas! The results:
java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: localhost; nested exception is: java.net.ConnectException: Connection refused]
With a Connection Refused error, we understand there is no service on the other side listening on that host/port (or there is service, but we have no way to access it). This means the service does not expose its port for us. For JMX we can utilize telnet and check to see if we have any connection to Cassandra via the JMX port which is 7199:
telnet localhost 7199 :~/tmp/java-docker$ telnet localhost 7199 Trying 127.0.0.1... telnet: Unable to connect to remote host: Connection refused
No connection. If we look at Cassandra's Dockerfile we see that it does expose port 7199:
# 7000: intra-node communication # 7001: TLS intra-node communication # 7199: JMX # 9042: CQL # 9160: thrift service EXPOSE 7000 7001 7199 9042 9160
However, it is not going to be exposed to an unlinked container or to our visualvm/jconsole from outside the container environment. For that purpose, when we run the Cassandra container we are going to use the -p option in order to expose that port also on the node itself. Our Docker run command would be as follows:
docker run --name containerized-cassandra -p 7199:7199 -d cassandra
When we run docker ps we can verify that the port was exposed:
$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 11b499d74c4e cassandra "/docker-entrypoint.s" 25 seconds ago Up 24 seconds 7000-7001/tcp, 9042/tcp, 0.0.0.0:7199->7199/tcp, 9160/tcp containerized-cassandra
You see that the only port exposed is the JMX port: 0.0.0.0:7199->7199/tcp.
If we now check to see if that port is exposed, we learn that it is:
$ telnet localhost 7199 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'.
Now let’s try again with jmxterm:
java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is: java.net.SocketException: Connection reset]
We receive a connection reset. Before, we received a connection refused because no port was exposed, and now we receive a connection reset. Let’s log into the image and try to understand why it happened.
$ docker exec -it containerized-cassandra bash
Let’s find the startup script to Cassandra and see where it configures JMX:
root@104f281f5f09:/# grep cassandra /etc/init.d/cassandra # Provides: cassandra NAME=cassandra CONFDIR=/etc/cassandra CASSANDRA_HOME=/usr/share/cassandra [ -e /usr/share/cassandra/apache-cassandra.jar ]||exit 0 [ -e /etc/cassandra/cassandra.yaml ]||exit 0 [ -e /etc/cassandra/cassandra-env.sh ]||exit 0 CMD_PATT="Dcassandra-pidfile=.*cassandra\.pid"
We see that Cassandra has a cassandra-env.sh. It’s used to configure the already started Cassandra instance. Let’s look at it and see how it configures the JMX:
root@104f281f5f09:/# more /etc/cassandra/cassandra-env.sh
…and we find this interesting code:
if["$LOCAL_JMX"="yes"]; then JVM_OPTS="$JVM_OPTS -Dcassandra.jmx.local.port=$JMX_PORT -XX:+DisableExplicitGC" else JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.port=$JMX_PORT" JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.rmi.port=$JMX_PORT" JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.ssl=false"
So, we learn if we do ps -ef | grep cassandra, we find that we have Dcassandra.jmx.local.port=7199, which means we are using the local.port and not setting the com.sun.management.jmxremote.rmi.port. This means that LOCAL_JMX is yes. Therefore, we are going to change LOCAL_JMX to no by means of passing an environment variable to our Cassandra Docker image.
$ docker run --rm --name containerized-cassandra -p 7199:7199 -e LOCAL_JMX='no' cassandra
Now we get Error: Password file not found: /etc/cassandra/jmxremote.password. Let’s see if the folder /etc/cassandra already exists in the container because, if it doesn't, we are free to mount it:
$ docker exec -it containerized-cassandra bash root@33acd467a75b:/# ls -la /etc/cassandra total 100 drwxrwxrwx 5 cassandra cassandra 4096 Feb 7 07:29. drwxr-xr-x 75 root root 4096 Feb 7 07:29.. -rw-r--r-- 1 cassandra cassandra 10200 Jan 7 21:41 cassandra-env.sh -rw-r--r-- 1 cassandra cassandra 1200 Jan 7 21:41 cassandra-rackdc.properties -rw-r--r-- 1 cassandra cassandra 1358 Jan 7 21:41 cassandra-topology.properties
There are multiple paths to take here:
Contribute to the original Cassandra image with an update specifying where the jmxremote.password exists.
Mount the /etc/cassandra externally with a volume to allow you to edit these configurations externally.
Extend the Cassandra image and add to it your own jmxremote.password file.
Connect the JMX passwords to your LDAP service.
While the LDAP service is our favorite option, as a repeating theme local mutations are handled by network accessible databases. For this example we are simply going to create our own image and add to it the jmxremote.password file.
We create a new jmxremote.password:
Our Dockerfile would appear as follows:
FROM cassandra COPY jmxremote.password /etc/cassandra/
$ docker build -t "mycassandra" Sending build context to Docker daemon 3.072 kB Step 1: FROM Cassandra ---> 368a7c448425 Step 2 : COPY jmxremote.password /etc/cassandra ---> 73a3a757047e Removing intermediate container ce8cf319d160 Successfully built 73a3a757047e
Run it with LOCAL_JMX=no:
$ docker run --rm --name containerized-cassandra -p 7199:7199 -e LOCAL_JMX='no' cassandra tomerb@TOMERBD_LAP:$ docker run --rm --name containerized-cassandra -p 7199:7199 -e LOCAL_JMX='no' mycassandra Error: Password file read access must be restricted: /etc/cassandra/jmxremote.password
So, we restrict the file access, Dockerfile:
FROM cassandra COPY jmxremote.password /etc/cassandra/ RUN chmod -R 600 /etc/cassandra/jmxremote.password
We again build the image:
$ docker build -t "mycassandra" .Sending build context to Docker daemon 3.072 kB Step 1 : FROM cassandra ---> 368a7c448425 Step 2 : COPY jmxremote.password /etc/cassandra/ ---> Using cache ---> fa43c7c67638 Step 3 : RUN chmod -R 600 /etc/cassandra/jmxremote.password ---> Using cache ---> 7f02e0aaf316 Successfully built 7f02e0aaf316
$ docker run --rm --name containerized-cassandra -p 7199:7199 -e LOCAL_JMX='no' mycassandra INFO 17:11:49 Starting listening for CQL clients on /0.0.0.0:9042 (unencrypted)... INFO 17:11:49 Not starting RPC server as requested. Use JMX (StorageService->startRPCServer()) or nodetool (enablethrift) to start it INFO 17:11:50 Scheduling approximate time-check task with a precision of 10 milliseconds INFO 17:11:50 Created default superuser role 'cassandra'
Now, log into JMX with the user we defined:
$ java -jar ~/Downloads/jmxterm-1.0-alpha-4-uber.jar --user controlRole --url localhost:7199 Authentication password: ******** Welcome to JMX terminal. Type "help"for available commands. $>
Success! We have connected to our containerized Cassandra JMX.