Fail Fast and Build Good Software

Preface

Our organization has concerns about code quality. Once or twice a year somebody from management reaches out with inquiries about the quality of the code our team had written. The boss may change but the question is always the same: “How good is your Sonar report?”

As a seasoned software developer, I should be used to this form of software quality management. Nevertheless, I immediately become disagreeable and tough. Firstly, I retort that Sonar is a good way to demonstrate hard work on software quality without actually doing anything on the matter. Then I go on with unflattering remarks about people who create problems, multiply problems, and store problems in a database. I finish with a statement that respectable software vendors do not waste efforts on defect databases; instead, they set up the process that prevents the occurrence of defects.

This is the subject of this article: the process that eliminates defects in a software product. We will implement the build infrastructure that complies with the Fail Fast principle. As an illustrative example, we will take a dummy software product written in two popular languages, Java and Scala, and we will use Maven as a build tool. As is tradition, you will find a link to the GitHub project at the end of the article.

Fail Fast

The idea is simple yet genius: stop the build process immediately if a problem is found. Don’t resume the process until the problem is resolved.

Today we know about the Stop the Line technique, thanks to the Toyota Production System. Every Toyota worker on the car assembly line was responsible for the product quality. If a worker noticed a quality problem, he activated an alert to indicate the problem. The production process was stopped until the quality issue has been corrected. The Stop the Line technique was adopted in Lean Software Development; maybe it is the best thing that the software industry ever borrowed from any other area.

According to the Toyota Production System (as well as to Lean Software Development), defects are a kind of waste that should be eliminated. Make a note: not stored in a database, not painted on dashboards, not included in “quality reports” – eliminated. Applying the Stop the Line principle to the software development process, we come to the idea of failing fast. In nutshell, the product cannot be built if a single quality problem exists. The build is failing, the assembly line is stopped, the red alert is flashing.

The Fail Fast approach allows us to locate a problem as early as possible. It provides the context of the problem, thus minimizing the correction efforts. In the end, the Fail Fast approach dramatically decreases the cost to fix a defect. What is more expensive – fixing a couple of lines in the code or delivering a patch to a production system? I am not even talking about financial penalties and reputational damage.

In the rest of this article, we will be exploring different ways to fail fast with the purpose of detecting problems early. We will embed quality assurance straight into the build process. We will practice implementing the build infrastructure that produces high-quality software.

Automatically Format the Code

Code formatting is not a matter of preferences but a part of the software quality. If the code is badly formatted, it quickly becomes unmaintainable. In the end, this means a high cost to locate and fix a defect or to add new functionality.

For a team of developers, code formatting rules should be common for all individuals. Of course, you can force every team member to configure his IDE with the same formatting rules. But new members join the team, others leave the team…no, this enforcement approach does not work well. Instead, code formatting should be a part of the product build infrastructure, it should not depend on the person running the build.

The Maven ecosystem provides formatting plugins for various languages. Java code can be formatted with formatter-maven-plugin. For Scala, we use scalariform-maven-plugin, which integrates a Scala formatting tool Scalariform with Maven.

<plugin>
  <groupId>net.revelc.code.formatter</groupId>
  <artifactId>formatter-maven-plugin</artifactId>
  <configuration>
    <configFile>java-formatter.xml</configFile>
    …
  </configuration>
  <dependencies>
    <dependency>
      <groupId>${project.groupId}</groupId>
      <artifactId>build-resources</artifactId>
    </dependency>
  </dependencies>
</plugin>

<plugin>
  <groupId>com.github.tashoyan</groupId>
  <artifactId>scalariform-maven-plugin</artifactId>
  <configuration>
  …
  </configuration>
</plugin>

As you may notice from the code snippet above, formatting-maven-plugin allows putting the formatting rules to an XML file (java-formatter.xml) and externalize these rules to a separate Maven artifact (build-resources). It is a good way to share common formatting rules across many projects.

We are not limited to Java and Scala languages, of course. There is a special plugin that formats Maven pom-files: sortpom-maven-plugin. Arbitrary XML resources can be formatted with xml-format-maven-plugin. In cases when there is no Maven formatting plugin for some language, we can use maven-antrun-plugin with our custom formatting tool.

Well, we intended to fail the build in case of a problem. How do we apply this approach to code formatting? Our product is built on the CI system. The code gets formatted during the build. If some developer hastily pushed unformatted code to the SCM system, then after the build is completed the working directory on the CI system will contain modified files. The CI system will be able to detect this situation by invoking an SCM command (for example, git status). If there are modified files in the working directory, the CI system will fail the build. The developer will learn to follow the good practice: don’t push the code without even trying to build it.

Put the Build environment Under Control

Now let’s explore the opportunities provided by the build tool we have chosen – Maven. First, we have to ensure that the build environment is correct: JDK has the required version, Maven itself meets the minimal version requirement. The maven-enforcer-plugin makes this job:

<plugin>
  <artifactId>maven-enforcer-plugin</artifactId>
  <executions>
    <execution>
      <id>Check build environment</id>
      <goals>
        <goal>enforce</goal>
      </goals>
      <configuration>
        <rules>
          <requireJavaVersion>
            <version>1.8</version>
          </requireJavaVersion>
          <requireMavenVersion>
            <version>3.2.3</version>
          </requireMavenVersion>
          <banDuplicatePomDependencyVersions />
        </rules>
      </configuration>
    </execution>
  </executions>
</plugin>

The maven-enforcer-plugin will fail the build if the environment does not match the configured requirements.

Note, that in the example above we also instruct Maven to check against duplicate declared dependencies: < banDuplicatePomDependencyVersions / >. However, we have a lot more work to do with respect to dependency management. Let’s stay focused on this problem in the following subsections.

Ban Unwanted Dependencies

Sometimes it is necessary to ensure that some unwanted dependency is not used in our product. Here is a real-life example: multiple logging frameworks. Suppose, our product has many third-party dependencies that use different logging libraries – Apache Commons Logging, Java Util Logging, Log4j, and maybe others. This is very inconvenient for the user who has to maintain logging configuration and log files for all these logging facilities. SLF4J provides a set of bridging libraries that converge the variety of logging libraries to SLF4J API. In order to bridge Apache Commons Logging to SLF4J, it is necessary to replace the commons-logging library with the SLF4J-provided jcl-over-slf4j library.

We need a build configuration that prevents commons-logging from occurring in the product classpath; otherwise, the bridging will not work correctly. Here is the configuration for the maven-enforcer-plugin:

<configuration>
  <rules>
    <bannedDependencies>
      <excludes>
        <exclude>commons-logging:commons-logging</exclude>
      </excludes>
      <message>Commons-logging should not be in the classpath. Use jcl-over-slf4j instead.</message>
    </bannedDependencies>
  </rules>
</configuration>

The configuration above allows Maven to fail the build if the unwanted library commons-logging comes as a transitive dependency. A developer has to fix this issue by means of Maven dependency exclusion.

Control Versions of Transitive Dependencies

A large project with many dependencies sooner or later comes to the problem of diverged transitive dependencies. For example, our product has two dependencies: A and B; dependency A requires SLF4J 1.7.15 and dependency B requires SLF4J 1.7.16. We have to ensure that our product uses strictly one version of SLF4J, and we have to manage the SLF4J version packaged within the product.

For a large product with dozens of direct dependencies and hundreds of transient dependencies, it becomes impossible to track diverged dependencies manually. The maven-enforcer-plugin provides a special rule to force dependency convergence:

<execution>
  <id>Check transitive dependencies are consistent</id>
  <goals>
    <goal>enforce</goal>
  </goals>
  <configuration>
    <rules>
      <DependencyConvergence />
    </rules>
  </configuration>
</execution>

When different versions of some dependency come to the Maven reactor, Maven fails the build. The developer has to explicitly specify the dependency version by means of dependency management. Thanks to the < DependencyConvergence / > rule, the developer always controls all dependency versions, thus the content of the assembled product package.

Prohibit Snapshot Dependencies for Releases

Once a product is switched to a release version, it is expected to depend on release libraries only. Here is a configuration for the maven-enforcer-plugin that bans snapshot dependencies for a release:

<execution>
  <id>Check no snapshot dependencies in release builds</id>
  <goals>
    <goal>enforce</goal>
  </goals>
  <configuration>
    <rules>
      <requireReleaseDeps>
        <message>Should use release versions in releases</message>
        <onlyWhenRelease>true</onlyWhenRelease>
      </requireReleaseDeps>
    </rules>
  </configuration>
</execution>

This is a good guard against incorrectly performed release procedure.

Respect Compiler Warnings

Before introducing advanced techniques of static code analysis, we can do one simple thing: take into account warnings from the compiler. Indeed, the compiler notifies about violations of basic programming practices. There is no much sense in complicated code analysis techniques if the compiler warnings are disregarded.

Here are javac arguments to be passed via maven-compiler-plugin configuration:

<compilerArgs>
  <arg>-Xlint:all</arg>
  <arg>-deprecation</arg>
  <arg>-Werror</arg>
</compilerArgs>

The following scalac arguments can be passed to scala-maven-plugin:

<args>
  <arg>-deprecation</arg>
  <arg>-feature</arg>
  <arg>-unchecked</arg>
  <arg>-explaintypes</arg>
  <arg>-Xfatal-warnings</arg>
  <arg>-Yno-adapted-args</arg>
  <arg>-Ywarn-dead-code</arg>
  <arg>-Ywarn-numeric-widen</arg>
  <arg>-Ywarn-unused:_</arg>
  <arg>-Ywarn-value-discard</arg>
  <arg>-Xlint:_</arg>
</args>

The key point here is that a compiler will fail the build in case of any warnings. Arguments like  -Werror  and  -Xfatal-warnings  will do the job.

Make Static Code Analysis a Part of The Build

Static code analysis is a set of tools that inform developers about problems found in their code. The list of detected problems may include violations of best coding practices, security breaches, performance problems and poor design. The only way to make static code analysis bringing any value is to run it as a part of the build process. In case of a single violation, the build should fail. Let’s make it clear: nobody cares about “reports” generated by static code analysis, and such reports are another kind of waste. Only the build failure produces the desired effect: the violation is eliminated immediately.

We will use the following tools for Java code: Checkstyle, FindBugs and PMD. For Scala code, we will employ Scalastyle. All these four tools provide plugins to integrate with Maven.

Here is an example of maven-checkstyle-plugin configuration:

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-checkstyle-plugin</artifactId>
  <version>${maven-checkstyle-plugin.version}</version>
  <configuration>
    <failOnViolation>true</failOnViolation>
    <violationSeverity>error</violationSeverity>
    <configLocation>java-checkstyle.xml</configLocation>
  </configuration>
  <dependencies>
    <dependency>
      <groupId>${project.groupId}</groupId>
      <artifactId>build-resources</artifactId>
      <version>${build-resources.version}</version>
    </dependency>
    <dependency>
      <groupId>com.puppycrawl.tools</groupId>
      <artifactId>checkstyle</artifactId>
      <version>${checkstyle.version}</version>
    </dependency>
  </dependencies>
</plugin>

The configuration for other static code analysis tools looks pretty much the same. The most important setting here is, of course, < failOnViolation>true >, it makes Checkstyle working in the Fail Fast mode. The set of Checkstyle rules java-checkstyle.xml is externalized to a separate module build-resources, so it can be reused in multiple projects.

One final remark on static code analysis, before advancing to the next topic. These tools provide checks to detect poor design. For example, Checkstyle has a rule that detects a too large number of classes referenced by some class. Such rules prevent developers from creating unmaintainable software. Sometimes managers reserve time for “code refactoring.” With rules against poor design, developers are forced to write well-structured code that often does not need refactoring in the future.

Don’t Disregard Unit Test Failures

Why do we ever write unit tests? Obviously, to let them fail. A test is expected to fail when it detects a problem. Then, why do we disregard test failures? No reason. We would better off stop writing tests – this would be more practical and honest.

For example, this is how we tell maven-surefire-plugin to not disregard test failures:

<plugin>
  <artifactId>maven-surefire-plugin</artifactId>
  <version>${maven-surefire-plugin.version}</version>
  <configuration>
    <testFailureIgnore>false</testFailureIgnore>
  </configuration>
</plugin>

For scalatest-maven-plugin, we also have to set the testFailureIgnore parameter to false.

Thoughts About Test Coverage

In my mind, test coverage is one of the most misleading concepts adopted by the software industry. Oftentimes, managers force R&D teams to measure the test coverage of the code and to increase it. They present the percentage increase to the upper management as some achievement in quality improvement.

First, how could it be that some piece of functionality has been delivered without tests? Suppose the measured test coverage is 30% — does this mean that two-thirds of the product’s functionality is not working at all? Then, there is no such work as “increase test coverage,” this is just “complete the work in progress.”

Second, qualified software developers often write tests before implementation. They write tests according to the specification and in parallel implement the functionality to make the tests passing. In other words, there is no such separate activity: to cover something with tests.

Furthermore, how many tests are need for a line of code? For a Java class? Certainly, it depends on the feature under test. We could estimate like this: at least one test for the success scenario, tests for invalid input, in addition, tests for 20% of unsuccessful scenarios occurring in 80% of cases. Fine, but this tells us nothing about the target tests number. Then, how could we define the target test coverage?

Needless to say, developers invent dirty tricks to make test coverage higher. Throughout my career I saw tests that do not test anything, those “tests” served as ballast to increase this meaningless percentage. A vicious circle: managers set meaningless KPI; developers imitate hard work on achieving these KPI; the software quality degrades; and managers set even more meaningless KPI. The total amount of efforts spend on this mockery may be higher than the workload needed to adopt the proper development process.

That is why I have a belief that test coverage is a Boolean property, that cannot be expressed as a percentage. If you make some product features without tests, then your test coverage is false. If you don’t test corner cases or unsuccessful scenarios, then your test coverage is false. And so on.

Explore Other Ways to Fail Fast

The Toyota Production System gave us the notion of Kaizen – the process of continuous improvement. As our product evolves, we have to detect more and more points on our assembly line where the product quality could be compromised.

Let’s consider a real-life example: XML samples for unit tests. Suppose, we are developing a web server that accepts requests from clients in XML format. We have implemented end-to-end unit tests, which start an instance of the server, send various XML samples to it, records the server responses and compares the actual responses with the expectations. We also have an XSD schema that specifies the client requests. The question is: are we sure that our XML samples comply with the XSD schema? If no, then we have a problem: our tests are based on invalid samples. Well, let’s validate all our XML samples during the build with help of xml-maven-plugin and secure our test infrastructure from malformed samples.

Virtually every project has some tools written in Shell dialects. With default settings, a Shell script does not stop its execution in case of an error. This behavior obscures problems and makes troubleshooting very difficult. Here are settings that make Bourne Shell scripts failing fast:

#!/bin/sh

set -o nounset
set -o errexit
set -o pipefail

SQL has the same disadvantage: an SQL script does not stop when a single SQL statement has failed. The fail-fast settings depend on the SQL dialect, for example, Oracle SQL*Plus provides the WHENEVER SQLERROR statement.

If your project has some automation tools written in Groovy, then use asserts and exceptions. If you automate deployment with Ansible, then consider the usage of the fail module and the any_error_fatal play option. And so on, and so forth.

Rethink Your Motivation

In this article, we explored various tools and techniques that help us to build great software. Now, my fellow developers, let’s think about our motivation. Why we are interested in applying these techniques? What pushes us to make the move towards the Fail Fast paradigm? Let me come up with the following statements:

• I build high quality software.

• I embed the software quality into the build infrastructure.

• I create the build infrastructure that quickly finds defects and empowers me and my colleagues to eliminate defects.

• I know my tools and use them properly to control the quality of the software I build.

• I help my organization to deliver excellent products and services.

Get Demo Code

Here a Demo Project on Github.

This project demonstrates all the techniques described in this article, as well as some additional features: creating a Linux rpm package and disclosing the information about open source components.

Links

Mary Poppendieck, Tom Poppendieck (2003). Lean Software Development: An Agile Toolkit

Bryan Helmkamp. Baruco 2013: Building a Culture of Quality