The First Drop: Introducing AdoptOpenJDK Quality Assurance (AQA)
AdoptOpenJDK Quality Assurance (AQA) is a curated set of open tests that we run against our release builds at AdoptOpenJDK.
Join the DZone community and get the full member experience.Join For Free
- Functionally correct.
You may also like: 5 Ways to Improve Your Quality Assurance
The tests are selected in the spirit of the AQA manifesto, the set of guidelines that governs our actions when testing OpenJDK releases. A brief summary of the manifesto is that tests should be:
- Open and transparent, both in code and the execution.
- Diverse and robust enough to cover the listed enterprise requirements.
- Evolving alongside implementations which means:
- Continual investment.
- A process to modify the set with oversight by the AdoptOpenJDK Technical Steering Committee (TSC), community involvement and quarterly reviews.
- Using code coverage data, comparative analysis, and other metrics to continuously improve.
- Portable enough to be run on developers' laptops or on any OpenJDK implementer's CI server
- Tagged/tracked and published so the exact test material can be found should we wish to rerun tests and reproduce results
AQA is not just a set of tests, but a system that keeps tests fresh and applicable using gathered metrics and input from the community. We want to “make quality certain to happen”, that is to assure quality for the community by being transparent, open and actively inviting community involvement.
Should you have feedback or input, please add comments to the AdoptOpenJDK/openjdk-tests AQA issue!
As I dive further into this topic, I will attempt to avoid too many water metaphors sprinkled throughout this introduction to AdoptOpenJDK Quality Assurance (AQA) ...or AQuA if you are so inclined.
"When the Well is Dry, We’ll Know the Worth of Water."
– Benjamin Franklin
Why AQA? We believe that open languages deserve open testing. When we looked at what testing was openly available to developers and to OpenJDK implementers and running on visible servers, the 'well' was somewhat 'dry'. There’s the OpenJDK regression suite, a wonderful set of tests, but does not cover all of the verification requirements needed to "make quality certain to happen".
Those regression tests begin to address the functional correctness requirement but we also needed to verify the security, scalability, performance, durability requirements. Our roadmap arose from the lack of a large and diverse pool of tests being used to verify the quality of OpenJDK binaries, and the desire to run testing in the open.
The AQA roadmap sees us expand open test coverage significantly. Our AQA roadmap sees us expand open test coverage significantly to include tests not only from the OpenJDK project but from many other open repositories. It includes open performance benchmarks, application suites (including Scala, Derby, Lucene-Solr, Jenkins, Wildfly, Kafka, Tomcat, and Elasticsearch), Microprofile TCKs, and a very large set of system and load tests.
Tests at AdoptOpenJDK are divided into groups:
- OpenJDK — The regression test suite from the OpenJDK project.
- System — System and load tests contributed to the AdoptOpenJDK and Eclipse OpenJ9 projects.
- External — Functional test suites from large Java applications plus Microprofile TCK tests, all run in containerized environments.
- Perf — Open-source performance benchmarks (both large-scale from dacapo and renaissance projects and microbenchmarks contributed to AdoptOpenJDK).
- Functional — Functional tests from both the Eclipse OpenJ9 project and those contributed to AdoptOpenJDK.
Each group is split into three levels:
- Sanity — Tests tagged as more critical to run frequently, testing against more commonly used packages or modules, or those areas that are under active development. These tests are run in nightly builds and release builds.
- Extended — Tests in this set are run less frequently (weekly), often targeting less used or less changed code.
- Special — Tests that need special hardware or configuration and/or can take an especially long time to run are in this test level. These tests are often excluded when running on a developer laptop or on limited hardware.
For version 1.0 of AQA, we selected tests that ensure our build and test mechanisms are working as we expect. Each of the 123 test targets in v1.0 contains many test cases. A few notes about the set:
- Sanity.openjdk = 8 targets, ~3300 test cases.
- Sanity.system and extended.system = 62 targets, ~25,000 tests. These system tests are repeated over many iterations to put the binaries under load, over 30 million times to properly stress them.
- Sanity.external = 4 targets, ~2500 test cases which are microprofile TCKs of various flavors (Openliberty, Thorntail, Payara, and Tomee).
- Special.functional = 45 targets that focus on Multi-Byte Character Set tests.
- Sanity.perf = 4 performance benchmarks (for v1.0, these targets are run and treated as functional tests, ensuring that they run without issue).
Additional details about the test targets can be found at the Graduated Testing and Test Numbers wiki page.
Multiplying the versions, platforms, and implementations expand the grid into a much larger test matrix. The implementations refer to different builds (J9=OpenJ9, HS=Hotspot, and Upstream respectively) which are available from adoptopenjdk.net.
Our goal and intention is for AQA to provide the community with a valuable quality assurance toolkit and set a high bar for OpenJDK binaries being produced and distributed. We believe that working together on this toolkit in an open and collaborative environment will be beneficial to all implementers, developers, consumers, and stakeholders within the open community.
Looking ahead to some of the specific work we will be initiated as part of AQA in 2020, much of it involves greater collaboration with the community.
We will bolster AQA performance testing by more closely collaborating with researchers and benchmark projects. Thanks to many wonderful sponsors, we will be able to secure some bare-metal machines on which to run benchmarks and effectively monitor the AdoptOpenJDK builds for performance regressions.
Through our collaboration with universities and researchers, we will continue to experiment to apply deep learning to our challenging OpenJDK verification problems, such as fuzzing compilers and (semi-automated) defect triage. Check out my presentation from the 2019 FOSDEM conference for a quick introduction to some of that work.
There are many great problems to solve, and many more curious and talented people in the community who want to get involved. 2020 promises to be an exceptionally exciting year for AQA evolution, as we bound forward as a community to verify OpenJDK binaries and support the development teams that make them possible.
Published at DZone with permission of Shelley Lambert. See the original article here.
Opinions expressed by DZone contributors are their own.