The State of Open Source Testing

The software testing community is certainly a passionate and opinionated bunch. Tour any forum or conference and you’re guaranteed to encounter a lively discussion. How much testing can (and should) be automated? How much should dev be contributing? Does unit testing matter? Shift left, right, or both?  

Yet, there is one thing that everyone agrees on: open source testing tools have revolutionized the profession and the industry.

Any tester working on web/mobile apps has at least researched—if not tried and adopted—Selenium, Appium, or a tool that builds upon them. And any testing tool vendor recognizes that it must address (and try to surpass) the value of these free and globally-esteemed tools in order to gain any traction in today’s highly-competitive market.

But, exactly how open source tools are impacting testing is not as clear. Open-source testing tools are used by a lot of different roles, in a lot of different regions, for a lot of different purposes. Are there any universal advantages—or challenges—to opting for open-source testing vs. commercial offerings? What nuances surface when you take a more targeted look at open source testing across functional testing vs load testing vs BDD? And how does it all vary across the globe?

The State of Open Source Testing project was designed to answer these questions. TestProject, SpecFlow, Flood, and Tricentis collaborated to capture a snapshot of where the industry stands today and track how open source testing trends evolve in the years to come. We were overwhelmed by the amount of interest in this project (we had nearly 2,000 respondents, making it the largest testing survey of the year), and we’re excited to share the results:

That’s just a teaser: see the complete infographic at the State of Open Source Testing page.

We’ll be taking a deep dive into the results—from several different angles—throughout the summer. To start, we wanted to highlight a few initial findings that jumped out to the report team.

It’s All About Cost…or Customization…Depending on Who You Ask

Overall, the top benefits reported for open source testing tools were as follows:

Cash is king. We believe that the current economic climate has placed increased focus on the cost saving benefits of open source technologies. Still, more than half of the users cited reasons beyond cost as their biggest realized benefits of using open source tools. A robust community, wealth of integration options, and limitless customization pushed open source tools above commercial options.

However, if you focus exclusively on the developer respondents, the front runner changes dramatically. A whopping 71% of developers cited ease of customization as their top reason for adopting open source testing tools. Across respondents with DevOps roles, community support and cost tied for the top spot.

Roadblocks: Technical Skills (But Not for Dev) and Support (for DevOps)

Technical skills remain the most valuable commodity, and a key to success in adopting open source technologies. Open source testing tools seem to be gaining support as viable options for even the largest and most security-focused enterprises, but some respondents still faced headwinds related to open source security and support.

Again, a different story emerges from the dev side. 65% of devs reported that existing commitments to another tool vendor is the main challenge to open source tool adoption. For the DevOps roles, technical skills and training required remained #1, but customer support and updates was a much greater challenge than it was overall (22% of DevOps roles cited it as the top obstacle vs 16% overall).  

Programming Languages: C/C++ for Devs, Java for the Rest

Java was the clear favorite in terms of programming languages. Given that it’s the default language to many popular open-source testing tools like JMeter, Selenium, and TestNG, this is no surprise. Runner ups were JavaScript and Python, which we believe lends itself to a wider audience of contributors with an accelerated learning curve.

Again, the developers were the outliers. C/C++ emerged as the most popular among developer respondents (four times more than Java).

Explore Trends Across Functional Testing, Load Testing, and BDD

This is just the tip of the iceberg. The report discovered tons of interesting findings across functional testing, load testing, and BDD. That’s why we assembled a panel of experts across each of those domains to take a deep dive into the results. Join us for additional data points, in-depth analysis, and some friendly debate, and watch the webinar here.