Spartan: A ''Forking'' Java Program Launcher, Part 3

The series so far: 

• Part 1

• Part 2

Requirements for Building Spartan

Spartan has been built and tested on RedHat/Centos distros of 6.7 and 7.4 (and Fedora 26, 27 as well as Ubuntu 14.04.5). Spartan consist of both Java and C++11 source code. It is built via the Java build tool, Maven. A Maven plugin is used to invoke cmake, which in turn compiles and links the C++ source code using GNU g++ compiler. Here are build provisioning prerequisites:

• Java SDK 1.8.0

• Maven 3.x.x

• cmake 2.8.11 through cmake 3.10.0

• g++ 4.8.4 (C++11) through g++ 7.2.1 (C++17) (the spartan code base is currently C++11 compliant)

• popt-devel C library (locate a package dialed in closely to your Linux distro version)

  • For RedHat/Centos 6.7 through 7.x:
    popt-devel-1.13-16.el7.x86_64.rpm

    • Refer to this repository to locate other rpm packages suitable to RedHat-style Linux distros:

      https://pkgs.org/download/popt-devel

  • For Debian-style distros:
    libpopt-dev_1.16-8ubuntu1_amd64.deb

    • Refer to this repository to locate other deb packages suitable to Debian/Ubuntu-syle Linux distros:
      https://pkgs.org/download/libpopt-dev

You might choose to have the following environment variables defined in your spartan build context, setting directory paths appropriately to match your installation (it's possible to build spartan on a 512 MB VM but you may have to reduce Maven max Java heap to, say, -Xmx208m, which then leaves sufficient memory for the cmake portion of the build):

export JAVA_HOME=/usr/local/java/jdk1.8.0_131
export MAVEN_HOME=/usr/local/apache-maven-3.3.9
export MAVEN_OPTS="-Xms256m -Xmx512m"

The Maven cmake plugin currently being used is:

<groupId>com.googlecode.cmake-maven-project</groupId>
<artifactId>cmake-maven-plugin</artifactId>
<version>3.7.2-b1</version>

By default this plugin will locate cmake at /usr/bin/cmake. If for your distro you have to install cmake, it might wind up placed at a different path. The GitHub project page of cmake-maven-plugin has information about customizing the plugin through configuration to use cmake at a different location:

Using a local CMake installation

Java Library Dependencies

Spartan makes use of these open source Java libraries:

<dependencies>
  <dependency>
    <groupId>org.javassist</groupId>
    <artifactId>javassist</artifactId>
    <version>3.20.0-GA</version>
  </dependency>
  <dependency>
    <groupId>ch.qos.logback</groupId>
    <artifactId>logback-classic</artifactId>
    <version>1.1.2</version>
  </dependency>
  <dependency>
    <groupId>ch.qos.logback</groupId>
    <artifactId>logback-core</artifactId>
    <version>1.1.2</version>
  </dependency>
</dependencies>

The Spartan Java API is made available per the Spartan.jar library, which will be located in the spartan installation directory. When doing a Maven build, use the install goal so that Spartan.jar will be installed in your Maven local repository, there it will be available for when building the spartan example programs.

Spartan Example Programs

There are three Spartan example programs located in the examples sub-directory:

• spartan-ex: illustrates all the Spartan annotations, also shows how to do a singleton worker child process sub-command, as well as how to use Spartan.invokeCommand().

• spartan-cfg-ex: illustrates how the supervisor service main thread can serialize a JCommander object instance to a file so that worker child process sub-commands can de‑serialize from that file in order to populate their JCommander object.

• spartan-watchdog-ex: illustrates how to code the supervisor to be a watchdog over some worker child process that it runs continuously.

Each is built using Maven, which produces a .jar file of the program in the respective target directory. The .jar file is what should be copied to a runtime directory; the spartan-cfg-ex program also requires the file examples/spartan-cfg-ex/target/config.properties and jcommander-1.72.jar to be copied to accompany it's .jar file.

How to Deploy and Run a Spartan Example Program

We will illustrate using spartan-cfg-ex. We will place all the programs discussed under the /opt/ directory (Wikipedia Filesystem Hierarchy Standard describes this as the location for optional or add-on software packages). The first step is to install the spartan launcher program.

Tip: In practice, because Spartan is intended for running Java programs as Linux services, it is recommended to create a spartanuser and spartangroup - deploy Spartan into a folder to where spartan:spartanis set as owner/group of the folder and all its files content. Then create a user account for running the service. This user account can be be added to the spartangroup and that group given execution permission of the spartan Java launcher program. Just add service user accounts to the spartangroup as any new services are stood up. The idea is to insure each service runs in a user account by which to control permissions/access, but then share the spartan executable between them for Java program launching.

Install spartan into an /opt/ sub-directory:

-rw-r--r-- 1 spartan spartan 9171 Jan 4 11:19 /opt/spartan/APACHE20-LICENSE.txt
-rw-r--r-- 1 spartan spartan 1510 Jan 4 11:19 /opt/spartan/BSD-LICENSE.txt
-rw-r--r-- 1 spartan spartan 193 Jan 4 11:20 /opt/spartan/config.ini
-rw-r--r-- 1 spartan spartan 750581 Apr 9 2016 /opt/spartan/javassist-3.20.0-GA.jar
-rwxr-xr-x 1 spartan spartan 218614 Jan 4 11:20 /opt/spartan/libspartan-shared.so
-rwxr-xr-x 1 spartan spartan 260801 Jan 4 11:20 /opt/spartan/spartan
-rw-r--r-- 1 spartan spartan 59015 Jan 4 11:20 /opt/spartan/Spartan.jar

For ease of running the example programs, execution permission is set for all users (normally it would be preferable if only owning user and group are granted execution permission):

sudo chmod a+x /opt/spartan/spartan /opt/spartan/libspartan-shared.so

Now create and populate an /opt/ sub-directory for spartan-cfg-ex:

lrwxrwxrwx. 1 my_user my_user 26 Jan 4 19:03 /opt/spartan-cfg-ex/spartan-cfg-ex -> /opt/spartan/spartan
-rw-r--r--. 1 my_user my_user 261 Jan 4 19:04 /opt/spartan-cfg-ex/config.ini
-rw-r--r--. 1 my_user my_user 249 Jan 4 19:06 /opt/spartan-cfg-ex/config.properties
-r--r--r--. 1 my_user my_user 69254 Jan 4 19:02 /opt/spartan-cfg-ex/jcommander-1.72.jar
-r--r--r--. 1 my_user my_user 12707 Jan 4 19:02 /opt/spartan-cfg-ex/spartan-cfg-ex.jar

Notice that there is a symbolic link spartan-cfg-ex which links to the spartan executable. This symbolic link and the config.ini file are the spartan conventions for launching a Java program.

When the symbolic link name is invoked as the program name of the service

The spartan program launcher will look in the directory where the symbolic link is located to find the file config.ini, where it will obtain various runtime options, such as JVM options for invoking the supervisor process. The config.ini file needs to be setup properly:

• Copy /opt/spartan/config.ini into directory /opt/spartan-cfg-ex/.

• Run chown to set the owner:group of the /opt/spartan-cfg-ex/config.ini file appropriately.

• Open config.ini in a text editor and edit the CommandLineArgs property.

• Given above install directories and file paths, the file should end up looking like so:

[JvmSettings]
CommandLineArgs=-server -Xms20m -Xmx50m -Djava.class.path="/opt/spartan/Spartan.jar:/opt/spartan-cfg-ex/spartan-cfg-ex.jar" -Djava.library.path=.
[ChildProcessSettings]
ChildProcessMaxCount=30
[LoggingSettings]
LoggingLevel=INFO

The logging level setting here is for the spartan program itself and when Java code calls the Spartan.log() API; the Java program will likely use logback, log4j, etc., for application logging in which case logging verbosity will be set through some other means for the general application logging.

Spartan currently only uses JAVA_HOME environment variable to locate the Java JVM shared library, so that will need to be defined appropriately in the runtime context of invoking the service.

If a user account has been established for executing the service, then the home directory could have been set as a default on that user account. However, it is possible to have multiple instances of a service being invoked to where each instance has its own home directory. The HOME environment variable can be defined in the context of invoking a particular instance of a service, in which case that will take precedence over any home directory associated to the service's user account.

Tip: Double check that JAVA_HOME and HOME have the values that are expected within the invoked execution context of the service.

Using the symbolic link name as the name of the program that is being run as a service has the nice effect that one can use that name with the Linux ps command to view its process status:

$ ps -C spartan-cfg-ex -Fww
UID PID PPID C SZ RSS PSR STIME TTY TIME CMD
my_user 31533 4301 0 30732 11240 0 03:24 pts/13 00:00:00 ./spartan-cfg-ex -service
my_user 31534 31533 2 494796 40132 0 03:24 pts/13 00:00:00 ./spartan-cfg-ex -service

The listing shows two processes but one is the parent launcher process that fork launched the supervisor process, which runs a Java JVM instance; the other is that supervisor process itself.

If a worker child process sub command has been invoked, then the ps listing might look something like:

$ ps -C spartan-cfg-ex -Fww
UID PID PPID C SZ RSS PSR STIME TTY TIME CMD
my_user 31533 4301 0 47116 11240 0 03:24 pts/13 00:00:00 ./spartan-cfg-ex -service
my_user 31534 31533 0 496845 43044 0 03:24 pts/13 00:00:01 ./spartan-cfg-ex -service
my_user 31595 4325 0 6109 3804 0 03:30 pts/14 00:00:00 ./spartan-cfg-ex cdcetl -run-forever some.json.gz
my_user 31596 31533 3 563971 35620 0 03:30 pts/13 00:00:00 ./spartan-cfg-ex -service

The process pid 31595, in this case, is Spartan invoked in client mode in order to handle launching a worker child process - the command line to the sub-command cdcetl is displayed here. The actual worker child process is pid 31596, which has another instantiated Java JVM instance. The -service option appearing on the JVM child process is just an artifact of the fork() call being utilized, but in actuality that child process received the command line arguments it was invoked with.

Presuming that the service is running foreground in one terminal console, then by opening another terminal, the command to see application status can be issued (notice the use of sudo to invoke the command as the same user the service is running as):

$ sudo -u my_user /opt/spartan-cfg-ex/spartan-cfg-ex status
spartan-cfg-ex: INFO: starting process 31639

*** timestamp *** | *** pid *** | *** command-line ***

2018-01-29T03:56:49.440 31596 "cdcetl" "-run-forever" "some.json.gz"

1 child processes active
spartan-cfg-ex: INFO: process 31639 exiting normally

The pid 31639 is spartan as invoked in client mode (the client mode process handles the output generated by the sub-command by echoing it to stdout).

Then to terminate the service, which will cause all processes to exit, can issue the supervisorstop sub-command:

$ sudo -u my_user /opt/spartan-cfg-ex/spartan-cfg-ex stop

As mentioned previously, the kill -TERM command can be used to cause a worker child process to exit without effecting the supervisor process of the service — just specify the child process pid number to the kill command as it is seen displayed in the status listing. Run the status command again and it will be seen that the child process has gone away.

This command line will cause a singleton worker child process to be launched:

$ sudo -u my_user /opt/spartan-cfg-ex/spartan-cfg-ex cdcetl -run-forever some.json.gz

Tip: Keep in mind that if file paths are passed as arguments to a sub-command, the user that the service runs as will need to have access permission established to that file. The example programs are not taking real files as arguments - they only serve for illustrating passing various arguments to sub commands.

A Spartan singleton refers to a worker child process sub-command that is restricted to being run as a single process at any given time. The Spartan.isFirstInstance() API is used to establish a fence around the core activity of the singleton sub-command.

The normal posture is that multiple instances of a worker child process sub-command can be invoked to run concurrently. The sub-command genetl is an example of a command that can be invoked to run many times concurrently; you can try it out by using the same command line as above but change cdcetl to genetl (you will have to start another terminal to run each sub-command, or else use a utility such as GNU Screen). All of these child processes will be listed by the status command — spartan keeps track of when these child processes are launched and when they terminate so the status stays updated automatically.

NOTE: Currently, the Spartan client mode does not respond to Control-C — the child process at the other end of the pipe it is reading from can instead be caused to terminate by using kill -TERMon its pid, which the client mode will sense as the pipe will indicate no more input to be read.