DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports
Events Video Library
Over 2 million developers have joined DZone. Join Today! Thanks for visiting DZone today,
Edit Profile Manage Email Subscriptions Moderation Admin Console How to Post to DZone Article Submission Guidelines
View Profile
Sign Out
Refcards
Trend Reports
Events
View Events Video Library
Zones
Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks

Integrating PostgreSQL Databases with ANF: Join this workshop to learn how to create a PostgreSQL server using Instaclustr’s managed service

Mobile Database Essentials: Assess data needs, storage requirements, and more when leveraging databases for cloud and edge applications.

Monitoring and Observability for LLMs: Datadog and Google Cloud discuss how to achieve optimal AI model performance.

Automated Testing: The latest on architecture, TDD, and the benefits of AI and low-code tools.

Related

  • Selecting the Right Automated Tests
  • Supercharging Productivity in Microservice Development With AI Tools
  • Mastering Node.js: The Ultimate Guide
  • Improving Customer-Facing App Quality Using Tricentis Testim

Trending

  • The Promise of Personal Data for Better Living
  • Harnessing the Power of In-Memory Databases: Unleashing Real-Time Data Processing
  • Essential Complexity Is the Developer's Unique Selling Point
  • Introduction to ESP32 for Beginners Using the Xedge32 Lua IDE
  1. DZone
  2. Testing, Deployment, and Maintenance
  3. Testing, Tools, and Frameworks
  4. Reducing Test Times by Only Running Impacted Tests

Reducing Test Times by Only Running Impacted Tests

Paul Hammant user avatar by
Paul Hammant
·
Jan. 25, 15 · Interview
Like (0)
Save
Tweet
Share
10.35K Views

Join the DZone community and get the full member experience.

Join For Free

This article follows on from my Omnipresent, Infallible, Omnipotent and Instantaneous Build Technologies one a couple of day ago. Specifically the last section: “Minimalist test execution, via hacks”, that addressed test times being very lengthy.

I’ve made a proof of concept of that for Maven. The tests impacted by a change (a pending change specifically) can now be quickly determined and fed into the test runner to massively reduce test times. Massively reduced, this is, if you’ve managed to engineer hours of Selenium tests.

Proof of concept

I forked a Github project that Petri Kainulainen had made to discuss coverage for a Maven projects.

Clone my fork, but cd into the “code-coverage-jacoco” directory.

Running testimpact.sh there – it will rebuild a map of what tests cover what production-classes (sources). I’ve checked in the previous results from running this script, that’s what a team would do. It runs Maven against one test at a time, to calculate coverage then store that per test. Actually for Petri’s example, I’m only focussing on the integration tests (run by the “failsafe” plugin) rather than the unit tests (run by the “surefire” plugin). Even though Petri’s example project is not launching Selenium (or equivalent slow test), the integration test phase is where you would run that for a canonical Maven project.

The testimpact.sh script uses python and ack (you’ll have to install those if you want to run this). I tried to use sqlite3’s CSV ingesting, but it was impossibly opaque, even with using StackOverflow’s best Questions/Answers, so I flipped to Python. Petri’s example uses JaCoCo for coverage, which spits out a handy CSV report (as well as HTML).

There are some text files in src/integration-test/meta/ccexample that show the sources covered by each test. Yes, they are checked in. Those files end in .java but are actually plain text (sorry):

There’s another file src/integration-test/impact-map.txt that contains a list of production sources and the tests that would exercise them. Actually its a map of sources vs tests:

Experimenting with what I’ve done

Change one of the two production classes in src/main/java/ccexample. Yes they are clones of each other – that’s just something I did after forking to increase the class count at the start. Don’t commit that pending change, just leave it there showing up as modified in git status.

Run python tests.py and watch it run one or two tests in the same invocation.

Now undo that change, and change the other source file, and run python tests.py again.

Different tests ran, right?

Undo that change, and do python tests.py once more.

No tests ran, right? That would be the same for changed sources/classes that had no tests exercising them (covering them under test invocation) at all.

Of course these few tests are really quick, but they could have been three subset from hundreds or thousands of tests with an elapsed time of many hours.

Turning the idea into a solution

  1. It is also worth noting that scripts as I have them are not robust or optimized.
  2. There’s more source-control systems than just Git, of course.
  3. The script needs to be able to work for a commit too, not just a pending commit.
  4. The storage format for impact-map.txt will not scale, and you might want to excise certain categories of POJO if every test exercises them.
  5. To be correct in Maven-land, this should be a bunch of plugins that fit the Maven style. One plugin would be invoked from Jenkins and would probably run constantly if corresponding test times are up in the hours. That Jenkins job should check in changes to the meta-files as it sees changes. This is benign, and useful to team members who may want the shorter build.

Would you check the impact map into source-control?

It seems to me that the map data is related to the other source files. Perhaps if you took a historical view to things, the impact maps change with the source code at the same time. If you were bug-fixing something from the past (checking out a prior revision), you might be happy that the impact map also goes back in time.

Of course there’s nothing here that couldn’t be stored in a key/value store, including the changing map over time. Or a service that could answer the question “what tests should I run if these source files are changed?” Except perhaps that uncommitted work (on your own workstation) isn’t going to be present in that store until after a commit, and the impact map data us updated to that into account.

Taking the idea even further

  1. I’ve suggested that Selenium is the technology that would greatly lengthen test times. There are many other test technologies that are one, two or three orders slower than perfect unit tests. This idea is applicable to much more than just Selenium – the only requirement is to be able to measure coverage while individual tests are running.
  2. What I have done could also be extended to the “surefire” phase – almost identical to what I’ve done already – an opportunity for reuse of course.
  3. CI daemons like Jenkins could benefit from the same impact-driven test time reduction. At least the per-commit jobs that we do for the Continuous Delivery era of enterprise development.
  4. This idea could be extended to the test-method level. That would be harder still, but achievable in the same way. It comes with arguably negligible gains, though. You’d code it all, and work out that 5% wasn’t worth it (versus other ways of speeding up tests).
  5. We’d need Intellij and Eclipse plugins for this.
  6. The script needs to be able to work for range of commits as many teams batch them in in Jenkins-land.

I wonder what is out there that already does this sort of thing already.

Followups (Jan 13, 17, 2015)

Markus Kohler notes:

This is a great idea and I guess similar to what Google does (there was blog post about it, can’t find it ATM).
But I think as it is it would not be completely correct.

The reason is that as far as I can see, you just run the tests that use the modified class files. That is not enough, because in Java changing one source code file might result in other files needed to be recompiled.

Gradle’s Java plugin (http://www.gradle.org/docs/current/userguide/java_plugin.html) computes that set of files for example for incremental compilation.

As an example if a final static String is changed and that String is used in other classes, these classes has to be recompiled because the Java compiler inlines these kind of strings. Any Test that uses these classes would have to be re-run.

He’s quite right there’s room for some edge case mistakes. To overcome those, you would need to check for changes that would lead to final static fields being inlined into other classes. Maybe a list of source files that, if changed, would cause all tests to be re-run (after a suitable warning).

unit test

Published at DZone with permission of Paul Hammant, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

Related

  • Selecting the Right Automated Tests
  • Supercharging Productivity in Microservice Development With AI Tools
  • Mastering Node.js: The Ultimate Guide
  • Improving Customer-Facing App Quality Using Tricentis Testim

Comments

Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 100
  • Nashville, TN 37211
  • support@dzone.com

Let's be friends: