Remote Debugging of Java Applications on OpenShift

in this article, i am going to show you how to attach a debugger and a visualvm profiler to the java application running on openshift. the approach described here doesn't make use of the jolokia bridge. instead, we are going to leverage the port-forwarding feature of openshift.

the whole setup can be divided into three steps:

1. enable debug and jmx ports on the jvm
2. set up port forwarding
3. attach debugger and visualvm to the forwarded ports

i am going to use openshift v3.11 that i installed using minishift and a test application built with java openjdk 1.8. this is how the complete setup is going to look like:

hello world application

for those of you who want to follow along, let's set up a test application, which we will use for debugging. if you already have your java application running on openshift, you can jump ahead to the next section.

let's deploy a hello world application that i found on github . this application was originally created to demonstrate how to build vert.x -based microservices on openshift. you can get this application up and running in just two steps.

first, issue this command to build an s2i builder image for vert.x applications:

$ oc create -f https://raw.githubusercontent.com/vert-x3/vertx-openshift-s2i/master/vertx-s2i-all.json
buildconfig.build.openshift.io/vertx-s2i created
imagestream.image.openshift.io/vertx-centos created
imagestream.image.openshift.io/vertx-s2i created
template.template.openshift.io/vertx-helloworld-maven created

openshift started the build of the builder image and you can follow the progress with:

$ oc log -f bc/vertx-s2i

...

removing intermediate container fc4bff8f426c
successfully built bd4a858867e9
pushing image 172.30.1.1:5000/myproject/vertx-s2i:latest ...
pushed 1/8 layers, 50% complete
pushed 2/8 layers, 25% complete
pushed 3/8 layers, 38% complete
pushed 4/8 layers, 50% complete
pushed 5/8 layers, 63% complete
pushed 6/8 layers, 97% complete
pushed 7/8 layers, 99% complete
pushed 8/8 layers, 100% complete
push successful

at the end of the build process, openshift pushed the new image into the integrated docker registry. next, we are going to use the builder image to build and run a sample vert.x application:

$ oc new-app vertx-helloworld-maven
--> deploying template "myproject/vertx-helloworld-maven" to project myproject

     vertx-helloworld-maven
     ---------
     sample vert.x application build with maven

     * with parameters:
        * application_name=hello-world
        * application_hostname=
        * git_uri=https://github.com/vert-x3/vertx-openshift-s2i.git
        * git_ref=master
        * context_dir=test/test-app-maven
        * app_options=
        * github_trigger_secret=em325a5k # generated
        * generic_trigger_secret=cbccciwr # generated

--> creating resources ...
    buildconfig.build.openshift.io "hello-world" created
    imagestream.image.openshift.io "hello-world" created
    deploymentconfig.apps.openshift.io "hello-world" created
    route.route.openshift.io "hello-world" created
    service "hello-world" created
--> success
    build scheduled, use 'oc logs -f bc/hello-world' to track its progress.
    access your application via route 'hello-world-myproject.192.168.42.115.nip.io'
    run 'oc status' to view your app.

you can follow the build logs by issuing the command:

$ oc log -f bc/hello-world

...

[info]
[info] --- maven-clean-plugin:2.5:clean (default-clean) @ vertx-hello-world ---
[info] deleting /opt/app-root/src/source/target
[info] ------------------------------------------------------------------------
[info] build success
[info] ------------------------------------------------------------------------
[info] total time:  1.102 s
[info] finished at: 2019-02-26t20:21:57z
[info] ------------------------------------------------------------------------
application jar file is located in /opt/openshift/vertx-app.jar
files located in the application directory:
total 13216
-rw-r--r--. 1 default root      286 feb 26 20:21 additional_files.md
-rw-r--r--. 1 default root 13525420 feb 26 20:21 vertx-app.jar
pushing image 172.30.1.1:5000/myproject2/hello-world:latest ...
pushed 0/9 layers, 2% complete
pushed 1/9 layers, 11% complete
push successful

if everything went fine, you should be able to see the hello world application running:

$ oc get pod | grep hello-world
hello-world-1-build   0/1       completed    0          6m
hello-world-1-dw5lf   1/1       running      0          42s

enabling debug and jmx ports on jvm

in the following, i am going to use openjdk 1.8. note that the available jvm options may vary depending on the version of the java platform you are using.

to enable a remote debug port on jvm, one has to pass the following option to the jvm:

-agentlib:jdwp=transport=dt_socket,server=y,address=8000,suspend=n

in order to enable jmx, the following jvm options are needed:

-dcom.sun.management.jmxremote=true
-dcom.sun.management.jmxremote.port=3000
-dcom.sun.management.jmxremote.rmi.port=3001
-djava.rmi.server.hostname=127.0.0.1
-dcom.sun.management.jmxremote.authenticate=false
-dcom.sun.management.jmxremote.ssl=false

this set of options deserves a bit more explanation. by default, jmx utilizes rmi as the underlying technology for the communication between the jmx client and the remote jvm. and as a matter of fact, there are two rmi ports needed for this communication:

• rmi registry port

• rmi server port

at the beginning, the client connects to the rmi registry on port 3000 and looks up the connection to the rmi server. after the successful lookup, the client initiates a second connection to the rmi server. based on our configuration, the client is going to connect to 127.0.0.1:3001. however, there's no rmi server running on the local machine, so what's the deal? as you will see in the next section, we are going to forward the local port 3001 back to the remote server.

next, we need to convey our configuration options to the jvm running inside the openshift pod. it turns out that there exists an environment variable java_tool_options that is interpreted directly by the jvm and where you can put your jvm configuration options. i recommend using this variable as there is a great chance that this variable will work no matter how deep in your wrapper scripts you are launching the jvm. go ahead and modify the deploymentconfig or pod descriptor of your application in openshift to add the java_tool_options variable. for example, you can open the deloymentconfig for editing like this:

$ oc edit dc hello-world

... and add the java_tool_options environment variable to the container section of the specification:

...

    spec:
      containers:
      - env:
        - name: java_tool_options
          value: -agentlib:jdwp=transport=dt_socket,server=y,address=8000,suspend=n
            -dcom.sun.management.jmxremote=true -dcom.sun.management.jmxremote.port=3000
            -dcom.sun.management.jmxremote.rmi.port=3001 -djava.rmi.server.hostname=127.0.0.1
            -dcom.sun.management.jmxremote.authenticate=false -dcom.sun.management.jmxremote.ssl=false

...

after applying the above changes, openshift will redeploy the application pod. at startup, jvm will print out the following line to the stderr which will show up in the container logs:

picked up java_tool_options: -agentlib:jdwp=transport=dt_socket,server=y,address=8000,suspend=n -dcom.sun.management.jmxremote=true -dcom.sun.management.jmxremote.port=3000 -dcom.sun.management.jmxremote.rmi.port=3001 -djava.rmi.server.hostname=127.0.0.1 -dcom.sun.management.jmxremote.authenticate=false -dcom.sun.management.jmxremote.ssl=false

this verifies that our jvm options are in effect and the debug port and jmx ports are open. how are we going to connect to these ports? let's set up port forwarding on the local machine next.

setting up port forwarding

openshift features port forwarding that allows you to connect to an arbitrary port of a pod running on openshift. port forwarding doesn't require you to define any additional objects like service or route to enable it. what you need though is to start a port forwarding proxy on your local machine. issue the following command on your local machine to start the proxy and forward the three ports 8000, 3000, and 3001 to the remote pod running on openshift:

$ oc port-forward <pod> 8000 3000 3001

in the above command, remember to replace <pod> with the name of your application pod. if everything worked well, you should see the following output:

$ oc port-forward hello-world-2-55zlq 8000 3000 3001
forwarding from 127.0.0.1:8000 -> 8000
forwarding from 127.0.0.1:3000 -> 3000
forwarding from 127.0.0.1:3001 -> 3001

note that the proxy keeps running on the foreground.

attaching to the jvm running on openshift

having our port-forwarding proxy all set, let's fire up a debugger and attach it to our application. note that we instruct the debugger to connect to the localhost on port 8000. this port is in turn forwarded to the port 8000 on the jvm:

$ jdb -connect com.sun.jdi.socketattach:hostname=localhost,port=8000

after the debugger attaches, you can list existing jvm threads using the threads command:

> threads
group system:
  (java.lang.ref.reference$referencehandler)0x133a                                             reference handler                                   cond. waiting
  (java.lang.ref.finalizer$finalizerthread)0x133b                                              finalizer                                           cond. waiting
  (java.lang.thread)0x133c                                                                     signal dispatcher                                   running
  (java.lang.thread)0x133d                                                                     rmi tcp accept-3001                                 running
  (java.lang.thread)0x133e                                                                     rmi tcp accept-3000                                 running
  (java.lang.thread)0x133f                                                                     rmi tcp accept-0                                    running
group main:
  (java.util.timerthread)0x1342                                                                vertx-blocked-thread-checker                        cond. waiting
  (io.vertx.core.impl.vertxthread)0x1343                                                       vert.x-worker-thread-0                              cond. waiting

...

next, let's check out if we can attach visualvm to our application as well:

$ visualvm --openjmx localhost:3000

works like a charm, doesn't it?

conclusion

in this blog post, we were able to attach a debugger and visualvm to the java application running on openshift. we didn't need to deploy jolokia proxy or create additional service or route objects to make our setup work. instead, we leveraged the port-forwarding feature already available in openshift. the demonstrated method has additional security benefits as we are not exposing any additional ports of the application container.

hope you enjoyed this article and was able to reproduce this setup for yourself. if you have any thoughts or questions feel free to add them to the comment section below.