Why You Probably Don't Use Static Analysis (and How to Change That)

Static code analysis can help achieve consistency and avoid bugs. While its benefits are supported by empirical evidence, it takes too much work to unlock them. Why?

• Setup overhead. Typically, this involves changing build scripts in many projects. Worse, this overhead is ongoing, as what's added needs to be maintained.
• Lots of instant work. When we set up static analysis tools, they immediately punish us by finding lots of problems. This creates work when we have other things to do.
• If the only hammer you have breaks the build, you use it sparingly. Traditionally, static analysis tools are applied in the build pipeline. There's a big disincentive to add static analysis to existing projects when it may immediately break the build.

My last post discussed auto-formatting. Static analysis provides another good example. Atomist can make introducing it painless, through:

• A consistent approach at a team level, avoiding duplication and drift over time.
• A ratcheting approach, where instead of immediate carnage, only new violations are flagged.
• An integrated UI such as Slack to surface issues without relying on build breakage.

What is ratcheting? In a recent Twitter conversation, microservices luminary Phil Calcado made an interesting point about checks on existing projects.

Something I've used to deal with this is by what's sometimes called "ratcheting". Assuming it is easier to tackle a issue (lint, test coverage, etc) when first introduced, make your pre check-in tests check that no new issues were added, but don't break the build bc existing issues

The initial wave of red is gone. We focus on improving quality going forward. The more we change, the more we improve.

Ratcheting Using Checkstyle and Atomist

Let's apply this using Checkstyle, a common Java tool. By default, Checkstyle will flag a lot of issues in a typical Java project, so eliminating legacy noise is important.

It's possible to create a Checkstyle suppression file, but it's a lot of manual work. Furthermore, baselining is a general problem, so it's best to handle it independently of Checkstyle.

Atomist to the rescue.

Atomist enables a consistent team-wide policy. Decide exactly what should happen, then make it so for all your projects.

To add Checkstyle across all projects, we use a code inspection goal. This can run on every push to analyze the state of a project. It will not block or break the build.

const checkstyleReviewer = checkstyleReviewerRegistration({
    checkstylePath: `${process.cwd()}/bin/checkstyle-8.8-all.jar`,
});
const sendMessageToUsers: ReviewListenerRegistration = {
    name: "messager",
    listener: async rii => {
        return rii.addressChannels(`There are ${rii.review.comments.length} issues`);
    },
};
const inspectGoal = withLegacyFiltering(
    new AutoCodeInspection()
        .with(checkstyleReviewer)
        .withListener(sendMessageToUsers));

The inspection goal is configured with reviewers. In this case, it includes out-of-the-box Checkstyle integration and listeners, which can route the resulting review output.

We can schedule this goal to run on any Java project via push rules:

sdm.withPushRules(
    whenPushSatisfies(IsJava).setGoals(inspectGoal),
);

Next, we need to add an Atomist extension pack to perform legacy filtering (ratcheting):

sdm.addExtensionPacks(legacyFiltering({inspectGoal, autofixGoal}));

Baselining

The Atomist goal uses the Checkstyle command line to perform the analysis. The checkstyleReviewerRegistration function comes out of the box with Atomist's Checkstyle extension pack.

Because we've enabled "legacy filtering," Atomist adds extra value that makes Checkstyle easier to adopt.

The first time it sees a push to a project, the inspect goal will execute a preliminary analysis to use as a baseline. It persists the result of that analysis in a file in the repo using Atomist's core autofix support. The following output shows what happens on the first review:

# spring-format-sdm Running Checkstyle on ... to establish baseline: file is .atomist/legacyIssues_Checkstyle.json

On subsequent pushes, Checkstyle will execute again, but the baselined violations will be ignored. However, any new violations will be reported.

Add Static Analysis to All Your Projects in Minutes

You can use this SDM locally or in the cloud, applying Checkstyle against all your Java projects. Simply clone https://github.com/atomist-blogs/spring-format-sdm, checkout the checkstyle branch, npm install and run atomist start --local.

When running against the cloud, you can have it raise issues, as well as report to Slack like this:

Further Possibilities

This approach allows baselining for any static analysis tool, allowing you to drop tools in at team level without being overwhelmed by immediate extra work. If it has an API or a command line, Atomist can run it on every push. Atomist integrates popular tools out of the box, and it's relatively straightforward to implement your own code inspections using them as a guide.

Applying inspections at a team level, rather than by modifying projects one by one, can help you find the ideal static analysis solution for your needs. It's just one way in which Atomist's team-wide policies can make your delivery both more consistent and more agile.