10 Biggest Mistakes in Using Static Analysis

Static analysis was introduced to the software engineering process for many important reasons. Developers use static analysis tools as part of the development and component testing process. The key aspect of static analysis is that the code (or another artifact) is not executed or run, but the tool itself is executed, and the input data to the tool provides us with the source code we are interested in. Static analysis started with compilers and derived technologies that are well established in the software development world. Each technology applicable for static analysis can choose between several alternatives, set up its own rules, and benefit from using them. What is most surprising to me is that even with a huge set of tools and possibilities, static analysis is not properly used and disregarded in most projects.

The List of Common Mistakes

#1 - A Violation Doesn’t Mean Failing the Build

Static analysis should help us to discover all the potential problems and raise a red flag whenever the codebase doesn’t meet the rules. The biggest mistake is to only produce warnings and not let the build fail when violations against the rules were thrown. This means the team will immediately start off ignoring any warnings from the system. When you are following a strategy where you can ship regardless of static analysis results, it’s bad, see mistake #10.

#2 - Rules Are Set by Architects and Developers Aren’t In the Process

Engagement of the developers themselves is essential for making static analysis effective. If the developers can’t affect the tools or are unable to cooperate with the rules, engagement will be slow or nonexistent. Developers are the ones who are taking care of code quality in everyday life, so you need to be friendly to allow them to participate in the rules, tools, and processes.

#3 - Not Set Standards From the Start of the Project

Setting up static code analysis and trying to remove problems after a decent amount of work on the project is done can be a nightmare. It’s so much easier to set up tools for checking the quality of the code at the start of the project and enforce them from the outset. For pet projects, it probably doesn’t make sense to set up these tools, but I have seen a pretty significant number of projects which started as pet projects and ended as regular projects. So part of the solution involves recognizing the right time for adding static analysis to the project.

#4 - Coding Standards Not Aligned With Static Analysis

If coding standards are speaking differently than rules in static analysis, that’s wrong and can be very misleading. These two guys should be playing the same game. Being careful about coding standards and having them aligned with static analysis is essential for communicating clear intentions about what’s useful for a project and what’s not. Also, lack of documentation about static analysis itself is to blame... things like how to:

• install it
• use it
• set up the build
• set up the continuous integration server
• mitigate the violations
• how/when to suppress violations
• handle legacy code - strategy for legacy
• and which parts of the software aren’t checked and why (don’t test what you can’t change).

This lack of documentation is a problem because it can easily break the engagement of team members and without it you cannot guarantee proper use of all the tools.

#5 - Tools Are Not Enhanced With Own Rules

Believe me, there is always something special in your software which should be checked as early as possible by static analysis.

#6 - Tools Are Not Complemented With Quality Assurance Tests

There are things which static analysis can’t discover... for example, configuration of the product or some specific runtime behavior. What you should provide is a good suite of quality assurance tests. These tests don’t test the functionality of the software, but they are ensuring that the codebase is following coding standards in a different way than what static analysis checks. It’s a common mistake to believe that static analysis is the only necessary tool to get rid of every problem.

#7 - Default Code Inspections Set in IDEs

When you are using some unified IDEs for developing software, you should provide the default project configuration for such an IDE (for example, settitngs.jar for the Intellij IDE). There is a list of code inspections set for almost every programming language and environment, and you should carefully set all the rules and disable/enable the rules according to coding standards and static analysis. Within every new rule in static analysis or coding standards, these settings needs be aligned as well.

#8 - Code Coverage Threshold Is Not Set or Regularly Increased

Code coverage is one of the tools from the static analysis toolset. It verifies that unit and integration tests check your production codebase. It’s a common mistake that there isn’t a set threshold which causes a failing build whenever the code coverage is under the threshold. And, it’s even worse to not increase this threshold regularly. It doesn’t need to be increased by some percents, but whenever we are increasing the code coverage, the threshold should be increased as well to ensure that new code added won’t be covered by the tests.

#9 - Complaints to False Positives

Yes, there are pretty obvious possibilities that analysis will throw false positives. You should take care about all the violations, and those which are false positives should be handled inside code with some suppression, using a comment to tell why there's a false positive and why we can safely ignore it.

#10 - When Adding Static Analysis, Starting With Too Many Rules

The fewer rules we are following the better. Adding static analysis to the product after some development is tough. But, not having static analysis on your side is worse. So, if you don't have sufficient time to incorporate a comprehensive set of tools and rules, start with some minimal number and work on improvements continuously. It’s even the standard for static analysis of greenfield projects—static analysis is a continuous process, and you should take care about that in a regular manner. Adding rules and avoiding inconvenient ones is the product of careful quality assurance.

Extra One - Lack of Management Buy-in

If you don't have management on your side, there is no simple way to have sufficient time for taking care with static analysis. It’s not a shortcut, but rather a long and difficult path. And a time-consuming one as well... so, you need management to have a solid understanding of the return on such an investment and convince them to allow you to set time, requirements, and goals for properly administering static analysis.

Conclusion

Static analysis is great. Using it is hard. Using it properly is very hard. With carefulness, sufficient time, and regular improvements any project will be rewarded with huge benefits: cleaner code, higher quality, fewer bugs, and better maintenance. Don’t make the mistakes listed above and you will surely see the benefits of static analysis.