Getting Started: Testing Concurrent Java Code
by
Join the DZone community and get the full member experience.
Join For Freei recently finished the last class of my master of science in computer science program at franklin university . i had to write a short paper for that class that i think is worth sharing with you. the paper was written with the class as the audience, so it’s a little simpler and lot less detailed than it should be. nonetheless, i think it has some merit to programmers who are trying to get started with testing their concurrent code. now that i’m done with school, i hope to explore this topic more fully, blogging the entire way. here’s the contents of the paper.
abstract
over the life of the java language, java developers have evolved their development life cycle into a complex and sophisticated ecosystem of tools, practices and conventions. rather than relying on gut feelings and verbal agreements that software has been tested just “good enough” to go to production, the modern java developer relies on metrics about the passing rate of numerous types of tests and code quality before deciding that code is production ready. unfortunately, this only applies to sequential java code—not concurrent code. despite having more complex metrics than sequential code, there is no excuse for excluding multithreaded java code from this modern development environment. this paper will help you to collect the tools and practices needed to modernize the way you develop concurrent java code.
acknowledgements
the author would like to thank venkat subramaniam and daniel hinojosa for their careful review of this paper and their thoughtful comments.
introduction
—the problem here is that programmers are not as scared of using threads as they should be.
david hovermeyer and william pugh
creating software that can be run by multiple threads concurrently is a daunting task—dwarfed only by the act of testing that code. nonetheless, concurrent code can be tested. however, to get to the topic of testing, some groundwork has to be laid out first. there are some contextual considerations that one must cover before being able to usefully test concurrent code. this paper will work through those fundamental concerns and how to address them in an effort to introduce how to get started with testing concurrent java code.
the modern java development model
there is no standard way to develop java programs. however, talk among amongst java developers, speeches by consultants and conference speakers and the writing of bloggers and professional authors all reverberate with common practices upheld by java programmers today. from this body of common knowledge you can glean what are the basic tools and processes used by the modern java developer.
the modern java developer does all of their development from an integrated development environment (ide) such as eclipse, netbeans or intellij idea. in the ide, they are able to perform quick and frequent refactorings (fowler) of their code which are enabled by unit tests that constantly verify the correctness of the changes. the modern java developer understands unit testing and, whether following the test driven development (beck) prescriptions or not, spends much of their coding time writing those tests. when the code is committed to source control, the more automated part of the process begins when some form of continuous integration (ci) (duvall, et. al.) system picks up the new code. a full compile and run of the entire unit test suite is run by the ci system, at a bare minimum, in order to continually verify the code. if they exist, more involved automated integration, functional, regression or user acceptance tests can be run to provide further verification.
aside from compilation and testing, the modern java developer relies on metrics. during the ci build it is typical to run a battery of tools against the code that gather metrics about its health. number of lines of source code counts, various static analysis tools and code coverage tools are all common. the developer understands how to interpret the output of these tools so their findings can be used to improve the code. improvements are implemented by making modifications; from simply adding missing unit tests to fixing latent bugs.
what is implicit in this collection of tools and practices is that only sequential java code is being reviewed. how to work with multithreaded code in this modern java development environment is never discussed. the remainder of this paper will help you begin to locate, test and verify multithreaded code in a way that fits into your role as a modern java developer.
finding concurrency bugs
you cannot test code if you are not aware of its existence. so before you can start developing tests for your multithreaded code, you must locate it in your code base. if you are writing the tests as you write that code, you don’t need this section. however, if you want to test existing code, you can use the tips here to help you hone in on the code you might want to test for multithreaded correctness.
locating multithreaded and related code
part of the reason that working with multithreaded java code is so difficult is that it is not trivial to figure out what code could be interacted with by multiple threads at runtime. start thinking about this problem by asking yourself the simple question “do i know where my multithreaded code is in my codebase?”—it’s highly likely that you cannot answer that question accurately. while it is a sensible idea to keep multithreaded code in isolation, you may find that your multithreaded code is sprinkled throughout your code base. there are two simple ways to get a much better idea of where that code is: by performing some simple text searches and by working with your peers.
there are several easy, yet fast and effective tools at your disposal that will seek out multithreaded code. the modern java developer’s ide is the most convenient to use. however, some simple command line tools can do just as well. in both cases, the onus is on the developer to determine what code could be subject to failure when being run simultaneously on multiple threads.
there is no tool in your ide that will list for you all the source code that might be run by multiple threads, so you have to devise a way to find that code. listing 1 is a collection of text strings that can be searched on that provides a good start to finding threaded code. all of the strings in the listing will uncover code that defines a thread, calls methods of the thread object or uses low-level locking mechanisms in the java concurrency api.
"implements runnable", "extends thread", "synchronized", ".notify()", ".notifyall()", ".wait()", ".wait(...)", ".interrupt()", ".interrupted()", ".join()", ".join(...)", ".sleep(...)", ".yield()", "import java.util.concurrent.atomic", "import java.util.concurrent.locks", “interruptedexception”
listing 1: text searches that help uncover concurrent code.
doing a simple text search for these strings in your ide will reveal much of the code that you are looking for [1] . if you prefer the command line, you can use the unix find command as in listing 2 to do a similar search.
find . -name *.java -exec grep -n -h --color "implements runnable" {} \;
listing 2: using the unix find command to search for concurrent code.
these searches can only yield so much, however. any java class is subject to being instantiated and used by a thread, which is not always as easily identifiable. for example, listing 3 demonstrates how any java class can be used in a thread. once you locate the class simplethread from the above search, you have to look into that class to identify what classes it uses since those referenced classes are now subject to multithreading concerns.
public class simplethread implements runnable {
private javaclass jc;
public void run() {
jc = new javaclass();
}
}
listing 3: a typical java class referenced from within a thread.
simple text-based searches are an easy way to start to identify what code may have to be tested, but they do not provide any information on the nature and intent of the code. for instance, some of the code may be known to the programmers to not need thread safety. for other parts of the code, it may be a total surprise that it could be run by multiple threads. given that, it is a good idea to implement a peer review process to analyze the severity of the need to test the found code for multithreaded safety (goetz). by having an experienced part of the team review the code, many important contextual details will be flushed out that are not evident to a single programmer. the findings from the peer review can be used to guide the testing strategy.
using static analysis to pinpoint concurrency bugs
understanding where concurrency exists in your code is a necessary precondition to testing that code. however, just locating the code does not identify bugs in it. so you need to progress from finding concurrent code to finding bugs in it and then testing to verify that it is bug free—all in a fashion that fits into your modern development environment. static analysis tools offer a way to achieve the middle step by identifying some bugs in your concurrent code. this section will discuss an open source static analysis tool called findbugs (hovemeyer & pugh).
findbugs is a reliable tool for picking out all sorts of “bug patterns” in java code. findbugs uses static analysis and heuristics to seek out code patterns and api usage that are indicative of or known to cause bugs (hovemeyer & pugh). the tool is effective enough, and sufficiently rid of false-positives, that it has been employed to look for bugs on production code at google (ayewah, et. al.). more pertinent to this discussion, however, is the fact that findbugs has the capability to find bugs in concurrent code. in fact, findbugs is the sole static analysis tool suggested by brian goetz for use with multithreaded code in the chapter “testing concurrent programs” in (goetz). that same chapter has a summary of the concurrent bug patterns that findbugs can identify, including inconsistent synchronization, unreleased locks, notification errors and spin loops.
goetz’s chapter on concurrent testing is out-of-date on one point. he praises tools like findbugs as being “effective enough to be a valuable addition to the testing process” but cautions that they “are still somewhat primitive (especially in their integration with development tools and lifecycle)”. goetz is referring to the fact that, in its infancy, findbugs was only available as a clunky java swing application. this is no longer the case as findbugs is now commonly integrated into the modern java developer’s tool set. as a basis, there is an ant task for findbugs that can be configured to produce xml output. further, several ci tools, such as hudson , jenkins , bamboo and sonar have plugins that can produce sophisticated dashboards from the findbugs xml output. the ant task allows for findbugs analysis to be run automatically as part of a build process where the xml output can then be picked up by the ci plugin to create the dashboard. the dashboards include all of findbugs findings, but make it simple to drill-down into just the concurrency bugs.
figure 1: the findbugs summary page in hudson.
findbugs is the second step in the chain of concurrency testing tools mainly because it is so simple to use. to use the ant task you just tell findbugs where to find the compiled .class files, where to locate the source code and where to write the output file. once the xml output has been produced, the ci plugins for findbugs generally only require as input a path to that file to work. the payoff for following such simple steps is huge. figure 1 shows a view of the findbugs dashboard plugin for the hudson ci server that lists the number of bugs found in the “multithreaded correctness” category [2] . figure 2 shows the dashboard’s ability to pinpoint the exact lines of code that contain the bugs. in this case, the class submitbrochureorder contains three concurrency bugs (mutable servlet fields).
figure 2: the findbugs dashboard showing bugs in the submitbrochureorder class.
measuring the amount of concurrent code tested
so far, this paper has covered two processes: locating what concurrent code exists in your code base and determining if your code contains any common concurrency bug patterns. another precursor to actually testing the code is to determine how much of the concurrent code is actually being tested with respect to thread safety. modern java developers are used to the measurement of “code coverage”(miller & maloney) and almost ubiquitously run both a battery of unit tests and code coverage analysis as a fundamental part of their build process. it is common to even base the health of a code base in part on the amount of code coverage. there are well known caveats with this practice (glover) yet it is still an effective way to get a sense for whether attention is paid to unit testing the code or not.
there is hardly an argument against the usefulness of code coverage analysis, but only for serial code. the typical code coverage analysis tools used by modern java developers, such as cobertura and emma , are not built to measure how well the concurrent aspects of the code are covered by unit tests. in fact, there is very little use currently of unit testing for concurrency, though goetz does briefly explain how the concurrent code in the java apis is unit tested (goetz).
to reconcile this problem, a team of researchers at ibm have developed the theory and tools to measure the extent to which concurrent code is tested for thread safety and have dubbed it “synchronization coverage”. this concept is not analogous to the code coverage metric for serial code. whereas serial code coverage measures the percentage of source code lines, branches, methods and classes that are executed during testing, synchronization coverage measures the percentage of critical sections of multithreaded code that are exercised by more than one thread concurrently during test runs. synchronization coverage is an umbrella term for multiple “coverage tasks”, each of which measures a different aspect of the thoroughness of the concurrent testing. as an example of a coverage task, if, while running tests, a synchronized block of code is not accessed by multiple threads concurrently, that block of code is not considered to have coverage. oppositely, if one or more threads has to contend for the lock on that critical section of code, the code is considered to be covered (bron, et. al.).
this metric can be quite revealing, especially if you are just starting to add tests to an existing code base. obtaining the synchronization coverage of your code base will immediately tell you if the thread-safety of your code is being tested at all by your existing test suite. since you now know where your concurrent code is located and have already weeded out easy-to-find concurrency bugs in it, you can use synchronization coverage to plan for what to actually start testing. the next section covers testing concurrent code, which includes using a tool from ibm contest that implements the synchronization coverage metric.
testing concurrent code
as with testing in general, there are many methods for testing concurrent code (watts). this section will focus on two specific modes of testing that fit particularly well in the modern java developer’s portfolio of tools. the two forms of testing are unit testing and automated exploratory testing. both types can be used to test threaded code to ensure that it works properly when run on multiple threads concurrently.
there are few options for unit testing concurrent code. however, one promising option is the multithreadedtc library(pugh & ayewah). this library uses the novel abstraction of a “metronome tick” to provide a mechanism for sequencing the interleaving of multiple threads. the library automatically moves to the next “tick” every time all the threads in a running multithreadedtc unit test are blocked. the tester can then assert that various conditions are held for a specific tick. the metronome tick allows multithreadedtc unit tests to verify the correctness of code’s multithreaded behavior in a way that does not interfere with the natural scheduling of threads by the jvm.
mutlithreadedtc is promising for two reasons. first, it is highly automatable, which fits perfectly into the modern java developer’s environment. multithreadedtc is built on junit, which makes it easy to run in an ant script as part of the ci build. this also means that it produces junit reports that can be viewed on the junit dashboard of a ci server. second, multithreadedtc is a rare tool in that it gives the developer the control to test very specific threading scenarios. any number of threads can be used in a test and the metronome tick allows the finest granularity of testing possible. this is a powerful and unique technique that does not exist in many other multithreaded testing tools.
in contrast, ibm contest is a no less useful and important tool that takes control away from the tester (edelstein, et. al.). rather than asking the tester to tediously code fine-grained threading scenarios, contest randomly explores as many of the thread interleavings as possible. the tester gets to write tests that more closely resemble serial unit tests, which contest then runs across a varying number of multiple threads simultaneously over a period of time. the idea behind contest is to help reduce the complexity of concurrency testing by covering as many threading scenarios as possible. in doing so, the chance of finding a concurrency bug increases.
contest is also highly automatable. it produces reports on the number of test passes and failures and even includes metrics on synchronization coverage. since it simply instruments your code, the developer can use any tests written for that code when running contest. in most modern java development ci systems, this amounts to adding two extra steps to the build process.
conclusion
modern java developers are comfortable with the cycle of developing code and tests simultaneously, building the code using a slew of tools for automation and then automatically verifying the code and producing metrics on the health of their code. metrics and verification are used as a means to constantly improve the quality of the code. even now, however, that entire process is focused on serial code only. it has been shown in this paper that developing tests for and verifying the quality of concurrent code can be integrated into this complex development environment as smoothly as the tools for serial code.
future work
it has been shown that there are more effective algorithms for finding bugs than what findbugs and ibm contest can provide (eytani, et. al.). however, these algorithms are not yet embedded in tools that are useful for developers outside of academia. some advanced industry tools (dern & tan) do make use of some of these algorithms, but there is little movement in this area in the java ecosystem. further, synchronization coverage is a straightforward and useful tool that could be integrated with serial code coverage seamlessly. there is, however, no tool known to the author other than ibm contest that implements synchronization coverage metrics for java code. it should be feasible, and is certainly desirable, that synchronization coverage be an additional part of existing code coverage analysis tools like cobertura and emma.
bibliography
ayewah, n., et. al. (2007). “using findbugs on production software”. oopsla’07, october 21–25. montréal, québec, canada.
beck, k. (2003). test-driven development by example . upper saddle river, nj: addison wesley.
bron, a., et. al. (2005). “applications of synchronization coverage”. ppopp’05, june 15–17. chicago, illinois, usa.
dern, c., tan, r. (2009). “code coverage for concurrency”. msdn magazine from http://msdn.microsoft.com/en-us/magazine/ee412257.aspx .
duvall, p., matyas, s., glover, a. (2007). continuous integration: improving software quality and reducing risk. upper saddle river, nj: addison wesley.
edelstein, o., et. al. (2008). automating the testing of multi-threaded java programs . ibm research laboratory in haifa
eytani, y. et. al. (2006). “towards a framework and a benchmark for testing tools for multi-threaded programs”. concurrency computat.: pract. exper. 2007; 19:267–279.
fowler, m. (1999). refactoring improving the design of existing code . upper saddle river, nj: addison-wesley professional.
glover, a. (january, 2006). “in pursuit of code quality: don’t be fooled by the coverage report”. ibm developerworks from http://www.ibm.com/developerworks/java/library/j-cq01316/ .
goetz, b., et. al. (2006). java concurrency in practice . upper saddle river, nj: addison-wesley.
hovemeyer, d., pugh, w. (2004). “finding bugs is easy”. oopsla’04, oct. 2428. vancouver, british columbia, canada.
miller, j., maloney, c. (february 1963). “systematic mistake analysis of digital computer programs”. communications of the acm. new york, ny, usa.
pugh, w., ayewah, n. (2007). unit testing concurrent software. ieee/acm international conference on automated software engineering, atlanta, ga, usa.
watts, n. (march, 2011). “a survey of methods and tools for testing parallel and concurrent programs”. written for comp 674 at franklin university.
[1] these strings will only reveal the most obvious multi-threaded code. there are other much trickier situations which have been left out of the scope of this paper (goetz).
[2] the dashboard has been created from the findbugs output for a real production code base developed at ohio mutual insurance group.
from http://thewonggei.wordpress.com/2011/07/18/getting-started-testing-concurrent-java-code/
Opinions expressed by DZone contributors are their own.
Comments