Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

Systemd, or How I Learned to Stop Worrying and Love Newness

DZone's Guide to

Systemd, or How I Learned to Stop Worrying and Love Newness

Systemd has become a part of many Linux distributions and, in my opinion, is here to stay. This post describes my encounter with systemd.

· DevOps Zone
Free Resource

The Nexus Suite is uniquely architected for a DevOps native world and creates value early in the development pipeline, provides precise contextual controls at every phase, and accelerates DevOps innovation with automation you can trust. Read how in this ebook.

Working in the IT world where things are not yet fully connected, integrated, and automated leaves a plenty of opportunities for engineers to get their hands dirty by hard DevOps work. Big Data Engineering is such a field of work where there is a lot of cogs lying around waiting to be integrated. In other words, DevOps and, in this particular case, Linux are inevitable and inseparable parts of Big Data Engineering. This is a story about one of those cogs.

Systemd is a relatively new addition to the Linux ecosystem. It is a system and service manager that is compatible with SysV and LSB init scripts and can work as a drop-in replacement for sysvinit. Systemd’s inception was followed by a huge controversy. The flame war is not yet fully extinguished, but systemd has become a part of many Linux distributions and, in my opinion, is here to stay. This post describes my encounter with systemd.

Prelude

The story started naively, as it usually does - another day, another DevOps task. This time, creating an Ansible role for deploying OS metrics, one of our tools at SmartCat Labs, was needed.

OS metrics is one of the goodies in SmartCat’s monitoring toolbox. It is an OS metrics data shipper or, in project terms, a reporter that sends collected data as a stream of events to Riemann server. More about the monitoring stack we use can be found in a separate blog post —  Monitoring stack for distributed systems.

So far, so good.

Plot

OS metrics is a python project. To start it manually, a python command like the following has to be executed.

$ python start.py -h <riemann server address>

As we want the OS metrics running as a system service, we need to provide an extra step to ‘daemonize’ it. The recommended way to do this in Ansible is to use the service module. It is an Ansible core module responsible for controlling services on remote hosts that supports a variety of init systems (BSD init, OpenRC, SysV, Solaris SMF, systemd, upstart). The service module provides a nice abstracted way of service handling. An Ansible task for starting could look like this:

- name: Start OS Metrics service
  service: name=osmetrics enabled=yes state=started

This is not enough to have a service up and running. Depending on the underlay init system, a service script or definition is needed. Even without a conciseness thought, the reflex was to provide a SysV init script. Old habits die slowly, and the /etc/init.d/ mantra is carved deeply into the gray matter. So, I extended the task like this:

- name: Install OS Metrics service init script
  template: src=osmetrics.j2 dest=/etc/init.d/osmetrics mode=664
  register: init-script

- name: Restart OS Metrics service
  service: name=osmetrics enabled=yes state=restarted
  when: init-script|changed

Then, I started looking for an init script, suitable for starting/stopping python process. One can have a favorite SysV init script, carefully tailored over the years and used in many situations (my precious!). I, on the other hand, belong to those that look for it out there every time. The thing is that you can never find a perfect one; it’s so elusive. You could end up with a SystemV init script template like this or this and boldly go where no man has gone before. But in the end it could resemble something like this (as it was in my case):

#!/bin/bash
#
# chkconfig: 35 90 12
# description: Foo server
#
# Get function from functions library
. /etc/init.d/functions
# Start the service FOO
start() {
        initlog -c "echo -n Starting FOO server: "
        /path/to/FOO &
        ### Create the lock file ###
        touch /var/lock/subsys/FOO
        success $"FOO server startup"
        echo
}
# Restart the service FOO
stop() {
        initlog -c "echo -n Stopping FOO server: "
        killproc FOO
        ### Now, delete the lock file ###
        rm -f /var/lock/subsys/FOO
        echo
}
### main logic ###
case "$1" in
  start)
        start
        ;;
  stop)
        stop
        ;;
  status)
        status FOO
        ;;
  restart|reload|condrestart)
        stop
        start
        ;;
  *)
        echo $"Usage: $0 {start|stop|restart|reload|status}"
        exit 1
esac
exit 0

Your mileage may vary depending on the distribution, distribution version, personal preferences and so on. But in the end, it looks very familiar and, what’s most important, it is simple and easily understandable in every of its incarnations. Right? In some cases, yes. In some other cases, you are supposed to meditate and live an ascetic life for several years in order to achieve the spiritual level needed for understanding it.

Once a decent init script was in place, I proceeded with the Ansible role using a CentOS 7 minimal Vagrant box as the test platform (CentOS 7 was the target platform). However, it started complaining right away that the service unit file had been changed and that systemctl daemon-reload has to be executed.

Newness

It didn’t take me long to realize that CentOS 7 is using systemd to manage system services. Even if interacting with the init V script directly, /etc/init.d/functions provides a way for systemd to hijack the process.

A close inspection of /etc/init.d/functions revealed that SYSTEMCTL_SKIP_REDIRECT variable can be used to avoid systemd completely. For example, you could add a line like this...

SYSTEMCTL_SKIP_REDIRECT=1

...into your init script before executing . /etc/init.d/functions and you should be good.

But I decided to embrace the newness and to give it a try to prove itself, despite all the horrible things that could be found googling for systemd. Most of these accusing posts were published in 2014 when systemd was introduced in some of the popular Linux distributions. Since then, systemd has gained a wider acceptance. Now there are plenty of overviews, documentation, and how-to resources. For example, this DigitalOcean community tutorial or RedHat System Administration Guide provide very good introductions on the subject, as well as basic usage instructions.

Unfolding

Systemd is a system and service manager for Linux operating systems. Except for daemons, it can control different system tasks such as file system mount points, timers, socket and D-Bus activation. Systemd introduced the concept of units (systemd units). Each unit represents a single systemd responsibility/task. The list of available unit types can be found here. These units are defined by unit configuration files. A systemd unit file consists of configuration directives that describe the unit and define its characteristics. Unit file names have the following form...

unit_name.unit_type

...and are located in /etc/systemd/system/. Unit files typically follow the following structure (a simple service unit example):

[Unit]
Description=human_readable_service_description
After=network.target

[Service]
ExecStart=start_command_or_path_to_executable
Type=simple
PIDFile=path_to_pidfile

[Install]
WantedBy=default.target

Where the unit file sections have the following functions:

  • [Unit] — contains generic options that are not dependent on the type of the unit. These options provide the unit's description, specify the unit’s behavior, and set dependencies to other units.
  • [Service] — this section, named after the unit type (hence [Service]), contains unit type-specific directives.
  • [Install] — information about unit installation options used by systemctl commands.

The above unit file is an example of a simple network-related service. The After specifies a list of services or targets that the service should be started after. ExecStart specifies commands or scripts to be executed when the unit is started. Type configures the unit process startup type that affects the functionality of ExecStart and related options. In this example, simple describes a service where the process started with ExecStart is the main process of the service. When systemd starts a system service, it stores the ID of its main process (PIDFile) in order to keep track of it. The systemctl utility then uses this PID to query and manage the service.

Knowing all this, an Ansible template for the osmetrics unit file (osmetrics.service.j2) could look like this:

[Unit]
Description=osmetrics
After=network.target

[Service]
ExecStart={{ osmetrics_install_dir }}/osmetrics
Type=simple
PIDFile=/var/run/osmetrics.pid

[Install]
WantedBy=default.target

where {{ osmetrics_install_dir }}/osmetrics is a convenient shell script for starting the osmetrics process:

#!/bin/sh

[ -f {{ osmetrics_config_dir }}/config ] && . {{ osmetrics_config_dir }}/config

/usr/bin/python {{ osmetrics_install_dir }}/start.py -h $OSMETRICS_RIEMANN_HOST -p $OSMETRICS_RIEMANN_PORT -i $OSMETRICS_INTERVAL

The osmetrics unit file should be placed in the /etc/systemd/system/ directory with correct file permissions using proper Ansible tasks:

- name: Install OS Metrics sytemd unit file
  copy: src=osmetrics.service.j2 dest=/etc/systemd/system/osmetrics.service mode=664
  register: result

- name: Ensure systemd is reloaded if osmetrics.service has changed
  shell: systemctl daemon-reload
  when: result|changed

- name: Restart OS Metrics service
  service: name=osmetrics enabled=yes state=restarted

And that’s pretty much it. Now compare the init V script with the osmetrics unit file! Pure zen, right?

Conclusion

In my opinion, the most significant benefit of introducing systemd in Linux is the regularization and unification of service-related administration tasks. For certain orthodox Linux users, it is precisely the differences between distributions that represent totems and tokens of superiority. If working with different Linux distributions on a daily basis is your job description, the systemd’s approach could really be a lifesaver.

Systemd promotes a declarative style (as an opposite of the imperative used in InitV, for example) for describing and controlling services. Avoiding imperative steps, that is, removing the need to describe a service through a series of commands that defines how and instead stating what needs to be done, reduces the complexity and makes things familiar and easy to maintain. For me, this is a proven way to ensure good maintainability and to reduce administration burden. Take the Ansible versus Puppet case, or Maven versus Ant, for example.

Replacing well established and old habits, such as the init system, can be hard enough by itself. Also, systemd is a relatively new project and, as such, is far from being flawless. But, having all its advantages in mind and its increasing adoption among major distributions, it seems well worth of exiting the comfort zone.

The DevOps Zone is brought to you in partnership with Sonatype Nexus.  See how the Nexus platform infuses precise open source component intelligence into the DevOps pipeline early, everywhere, and at scale. Read how in this ebook

Topics:
devops ,systemd ,python ,big data engineering

Published at DZone with permission of NikolaIvancevic, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}