Spartan: A ''Forking'' Java Program Launcher, Part 1

From Wikipedia: Fork (system call)

In computing, particularly in the context of the Unix operating system and its workalikes, fork is an operation whereby a process creates a copy of itself. It is usually a system call, implemented in the kernel. Fork is the primary (and historically, only) method of process creation on Unix-like operating systems.

Introduction

The Java program launcher, java on Posix platforms and java.exe on Windows, has remained pretty much the same in functionality since the origin of the Java programming language. This default Java program launcher provides a uniform experience for starting the execution of Java programs on all supported platforms.

But after two decades the facility for launching Java programs is due for a face lift.

• Today the predominate use of the Java language is in writing backend or middleware, server-based software, and in more recent times the interest has shifted to centering on implementing microservices (Java is also popularly used for Android mobile device computing but is not relevant to the discussion here).

• These days Linux (or Posix) operating systems rule in the realm of server-based software - this is principally because cloud computing data centers mostly rely on a Linux-based OS as the host for their VMs. Also, enterprise computing looking to homogenize development across on-premise data centers and cloud-based computing have gravitated toward Linux as a common denominator.

• Containerization coupled with microservices are dominate trends in server software development and its subsequent deployment. Linux is the most prominent OS platform for containerization because the most widely used container implementations are based on Linux kernel features.

• Java threads are problematic when writing high availability, self-healing software - whereas the process of modern operating systems represents a much more robust unit of program execution that can be killed with impunity and then the program relaunched (Java threads - well, not so much).

• SSH-based command-line shell connections are a pervasive means of connecting to the server host. The ability to interact with services software from a command line can be a nice enabler for both developers and support staff - particularly if the service can respond to sub commands initiated from the command line and do so with practically no special effort required by the programmers of the service. Also, having such command line interaction capability with a running service daemon that is not dependent on TCP sockets insures a tighter default security posture.

In view of these factors it is time to take a new look at the matter of launching Java programs. Why not devise a new kind of Java program launcher that is better fitted to the above landscape of contemporary computing?

And so such a program has been written (implemented in C++11) and it is called Spartan.

What Is Spartan?

The Linux native program Spartan is an alternative Java program launcher. It is specifically intended to launch a Java program to run as a Linux service daemon - which means the Java program stays resident indefinitely until issued a command instructing it to exit.

The special command line option -service is reserved by Spartan and instructs the program to be run as a service daemon (yes, it overlaps with the Java JVM -service option but we will see how JVM options are dealt with later).

A Spartan annotation is required to designate the main() method entry point for the service daemon to begin execution at.

A Spartan-based Java program will respond to two sub-commands without a programmer doing anything special:

• status 

• stop 

The method implementing status sub-command can be easily overridden and customized (to display more in-depth status info).

A Spartan-launched program can also be terminated via Control-C (Posix SIGINT signal).

Additionally, Spartan annotations can be applied to methods that are designated as entry points for programmer-defined custom sub-commands.

Supervisor Process and Worker Child Processes

The Spartan service daemon runs in a process context referred to as the supervisor process. It is able to easily launch worker child processes that execute from the very same Java program code as the supervisor.

Spartan annotations are also used to designate method entry points for worker child processes.

A sub-command can therefore be issued to invoke a worker child process from the command line (in addition to supervisor-specific sub commands already mentioned).

The spartan annotations for supervisor sub commands and worker child process sub-commands will specify the sub-command text token as an annotation attribute. That token is what is typed at the command line in order to execute it - along with any command line arguments that are passed to the sub-command. Consequently, Spartan-based services are very easy to script via Linux shell programs such as bash.

A supervisor process can oversee the execution of one or more worker child processes - it can supervise them, so to speak. This is a great way to go about programming robust, self-healing software.

Multiple worker child processes could also be established with an arity corresponding to the count of detected CPU cores - the supervisor process could oversee the dispensing of work to be done by these child processes, so yet another way to partition a program for parallel processing (JVM heap size is kept to more minimal level per each parallel processing activity; the GC of each child process will be less stressed and perhaps faster due to smaller scoped garbage collections).

Native Java serialization can be effortlessly used to communicate data between these processes - they're all running from the same Java code so the classes are the same, and thus no version compatibility issues to contend with. Java serialization is also the simplest way for the supervisor to convey one-time configuration to any worker child process.

The Linux security model of the supervisor is the same for the worker child processes.

A Spartan-based program by default does not use any TCP socket listeners - it is up to programmers to decide whether to introduce the use of any socket listeners, as determined by the domain requirements of their particular program.

Spartan enables straightforward multi-process programming for Java programmers - read on to see why Spartan exceeds the existing Java process-related APIs in the, all crucial, ease of use department.

Why Processes Instead of Threads Only?

Threading came about as an evolutionary step beyond the operating system process as a unit of execution - a thread is a more light weight construct from a scheduler context switching perspective and thus multiple threads can be executed within a single process context (threads exist in the owning process address space). However, when designing software for high availability rigor, with self-healing capability, the thread is dismal by comparison to the process as it can too easily become catastrophically unstable and thereby render the entire Java JVM unstable.

The Java JVM runs in a single process and only has threading available by which to manage concurrent activities. When any one JVM thread becomes destabilized, the entire JVM is highly prone to being compromised too. Java thread APIs destroy, stop, suspend, resume, these are all deprecated. Here is an excerpt from the deprecated Thread.stop API:

Deprecated. This method is inherently unsafe. Stopping a thread with Thread.stop causes it to unlock all of the monitors that it has locked (as a natural consequence of the unchecked ThreadDeath exception propagating up the stack). If any of the objects previously protected by these monitors were in an inconsistent state, the damaged objects become visible to other threads, potentially resulting in arbitrary behavior.

A Java thread cannot really be killed explicitly without high potential for undesirable side effects. As such, Java threads must be coded to terminate nicely under their own volition. If a thread becomes wedged, though, then the running service is out of luck - not much can be done for that situation. This is not the ideal for when tasked with creating software systems where high availability and self recovery are paramount concerns.

However, it is more straight forward to write robust high availability software where one process watches over another process in which the real work is being done. This arrangement is referred to as a watchdog. The watchdog parent process health check monitors the subsidiary child process - if the child process encounters error conditions or even out-right crashes, then the parent watchdog can take self-healing remediation actions that eventually may restore operations to normal. A wedged process or a fatally crashed child process can be explicitly killed by the parent watchdog process (or if the child process terminated abruptly the parent can detect that). A new child process can then be launched by the parent watchdog process.

In such error handling situations, the cycle might continue for a while - say, if a database resource had gone off line for servicing but later was brought back online, in which case a worker child process then successfully gets a connection to the database resource and resumes operation.

Or the error condition may continue and require support staff remediation. In that event the watchdog process remains in a stable condition to where it can be monitoring the number of retries before it decides to communicate alert status to support staff. As the error condition persists, the watchdog can apply back-off heuristics and perhaps spam suppression of alerts.

A watchdog coded specifically to the needs of a service can finesse the understanding of possible error conditions that might arise much more so than generic mechanisms that are typical of monitoring software packages. Having application specific knowledge can be very helpful as to error context - when retry is warranted vs. an out-right fatal condition requiring alerting, etc.

A custom implemented watchdog is the ideal because it can provide the smartest set of reactions to potential error conditions that a particular service may encounter - because it was written by a software developer that has the highest domain knowledge about the service.

Process Forking: The Easiest and Best Way to Implement a Watchdog

When programming in C or C++ (or some of the new languages such as Rust) on a Posix OS, it is possible to use the fork system call to spawn a child process from a parent process. This is a highly convenient mechanism for concurrent programming via processes because one can easily write the logic of the parent process and the child process(es) to reside all in single program, where said program is one executable image.

A single program using process forking will start out executing as the parent watchdog process. After making the fork call, it will then continue executing as the parent process along one code pathway, and then as the child process along another code pathway. It is easy to thus share the programming use of the same data structures and functions between parent and child processes.

The marshaling of data between parent and child processes will be guaranteed to be version compatible (the exact same data structures are seen identically by both) so there is no need to worry with version checks in parent-child inter-process communication.

The child process inherits and executes with the same permissions as the parent process so it can automatically access all the same files and directories that the parent can access. By the same token, the child is likewise prohibited from unwarranted resource access just as the parent. Hence the security posture is simple and straightforward - the child process can be regarded in the same manner, security-wise, as the parent process. Because of this inheriting of user identity and permissions, it is not necessary to provide authentication credentials when initiating the child process via a fork call.

Well, that is all nice and wonderful for C, C++, or Rust programmers, but what does that have to do with Java programming?

A core purpose of the Spartan Java program launcher is to enable a manner of programming with Java that closely resembles process forking goodness. A Spartan programmer really does Java programming with processes but in a manner that retains the benefits just described.

Yes, Java already has APIs for launching other processes - but at a cost relative to fork call convenience and simplicity. Sometimes a major step forward in programming is a matter of making something much easier to do, with much reduction of the negative drawbacks, worries, concerns. Because of the latter, often times a — such as the Java API for launching processes — can go virtually unused. There is usually just too much of a hassle factor involved so programmers by and large just don't bother to go there.

What Spartan Brings to the Table

A Java program launched via Spartan has these capabilities and characteristics:

• Is intended for implementing services - invoking main(...) initiates the program to run as a service daemon.

• The service runs as the supervisor process.

• A supervisor process can easily initiate a worker child process, which executes from the same executable program as the supervisor.

• Supervisor and worker child processes thus programmatically share all the same classes and methods.

• A worker child process executes as the same user and has the same permissions as the supervisor.

• A worker child process can i/o communicate to the supervisor via a Java java.io.InputStream pipe.

• The supervisor can easily kill a child process and start another.

• The supervisor can easily be aware of when child processes terminate (by virtue of reading from their InputStream pipe).

• Supervisor can launch multiple child processes which can be associated to initiate their execution from different class method entry points to suit different purposes.

• Sub-commands can be invoked from a command line shell (e.g. bash) where they are trivially handled in the context of the supervisor process, or result in a child process being launched to carry out the sub-command.

• Supervisor has automatic support of a status sub-command — this will, at a minimum, list any active child processes but can be custom overridden for enhanced status info.

• Supervisor has automatic support of a stop sub-command which causes the running service to cleanly exit (supervisor and any worker child processes all quit).

• Spartan insures the supervisor is kept aware of child processes being started and terminated so as to keep status info current.

• Spartan really does use the fork system call when establishing the supervisor process and any child processes, hence Spartan is currently only available on Linux.

• Each such process initializes its own instance of the Java JVM (a single instantiated Java JVM does not support being forked).

• The easiest way to share configuration state is for the supervisor to Java-serialize a configuration object to a file and then child processes deserialize that config object when starting up; in this way, startup configuration processing is done just once by the service supervisor process.

• Of course, Spartan programs can be coded to dynamically retrieve config info from services such as Consul too, or coded to use a combination of startup config initialization combined with point-in-time dynamic config retrieval.

• Spartan makes use of custom annotations to denote:

  • supervisor main entry point.

  • supervisor sub-command entry points.

  • worker child process sub-command entry points.

• Spartan has the requirement that the owning class of the main method entry point be derived from spartan.SpartanBase.

• The program name (as used to launch the service) is available to the supervisor main method entry point thread in the spartan.SpartanBase.programName static field and via its  getProgramName() static getter method.

• The class spartan.SpartanSysLogAppender is derived from ch.qos.logback.core.OutputStreamAppender<ILoggingEvent> and enables Java code to use this logback appender to log error and fatal messages directly to the Linux syslog (no TCP port is involved - Spartan directly calls Linux API for syslogging).

• The Spartan convention is to use a symbolic link (having the desired program name) reference the spartan program launcher executable.

• Spartan will look for a config.ini file in the same directory as the symbolic link.

• The config.ini can contain settings such as Java JVM settings that instantiate the JVM for the supervisor process.