Two Vital Components Your Continuous Delivery Process Can’t Ignore

For many enterprises, Continuous Delivery (CD) processes have become a ‘necessity’ rather than a ‘luxury.’ The value is certainly clear: CD can help IT teams lower risks while working leaner, faster, and with more agility. Time-to-market is accelerated and internal headaches are minimized.

But even if you’re already running a Continuous Delivery process, are you using the right tools to execute it most effectively? Let’s take a look at how Continuous Testing and Diagnostics tools work with CD, and how to integrate BlazeMeter and New Relic into your CD processes.

Why Continuous Testing and Diagnostics Are Key

You can’t run an effective Continuous Delivery process without Continuous Testing. Continuous Delivery can reduce time-to-release from several months to just a few hours. However, without the right testing, your CD process may be destined to fail. As your software evolves, you’ll need to run more and more tests but, if you’re like most companies, you’ll be able to execute only a tiny fraction of these. You could be faced with software quality compromises or a longer time-to-release.

So how can you avoid this? A truly agile process, as championed by Continuous Delivery, demands complete automation—and that includes the testing. Automated testing allows your processes to run faster and smoother. For it to be truly effective, this automated testing should be continuously integrated throughout the Systems Development Life Cycle (SDLC). This will allow you to catch and prevent performance issues before they happen—whatever stage of the life cycle you’re in.

That’s why we believe that continuous, automated testing integrated throughout the development life cycle is key to the success of your Continuous Delivery.

What Role Do Diagnostics Play?

Put simply, Continuous Testing and Diagnostics go hand in hand.

Testing tells you one side of the story. It can show you when you’re going to have problems, the precise point at which you’re going to hit that bottleneck. However, it doesn’t show you where you’ve hit that bottleneck or pinpoint the area of your application causing the problem.

Due to the increasing complexity of applications, pinpointing the cause is harder than ever. Most modern applications don’t consist of a single server. Instead, there are several layers of architecture and different roles for the servers (app servers, backends, storage and database servers, etc.). What’s more, some of these servers are located in the cloud while some are on premise. Finding the root cause can be extremely complex under such conditions.

To truly resolve the issue, you need to see the full picture. This means testing to pinpoint where you’ll hit that bottleneck and using a diagnostics tool to uncover the root cause of the problem. Using testing and diagnostics together gives you everything you need to fix the problem and move on to the next step in the Continuous Delivery process.

How to Implement Continuous Testing and Diagnostics Into Your CD Process

As mentioned, an efficient, agile process typically requires automation. As such, it’s best to use automated tools for your testing and diagnostics. Let’s look at the combination ofBlazeMeter’s Continuous Testing Platform and New Relic APM and give some practical examples on how you can employ them at different stages of your software life cycle.

BlazeMeter is designed to integrate seamlessly with various Continuous Integration, Continuous Delivery, and Source Control tools to provide continuous automated testing throughout the software life cycle. On the other side of the coin, New Relic enables you to continuously diagnose your application internals—in preproduction and postproduction.

For example, let’s look at a typical development workflow that incorporates continuous testing. A developer made a code change that caused the response time for an API call to increase from 200ms to 300ms, which is more than the 250ms threshold defined for testing that API call. The code looks fine on his box, so they do the following:

1. Commit the code into a Github repository…
2. The Github commit triggers a Jenkins build…
3. The Jenkins build launches a process that pushes the latest code to a baseline testing machine…
4. Jenkins uses the BlazeMeter plugin to run a load test against the baseline machine…
5. The BlazeMeter plugin fails the build due to a threshold failing: 250ms for the API call…
6. The developer sees the failed build in Jenkins, and clicks on a link to the BlazeMeter report from within the Jenkins build report…
7. The developer also checks the New Relic report to see where in the code the issue is…
8. The Developer fixes the code and commits again…
9. The Github commit triggers a Jenkins build…
10. Jenkins launches a process that pushes the latest code to a baseline testing machine…
11. Jenkins uses the BlazeMeter plugin to run a load test on the baseline machine…
12. Test passes…
13. Everyone is happy

Now let’s look at how we can use these tools.

Testing and Diagnostics in Preproduction

In preproduction, it’s important to test various external Key Performance Indicators (KPIs), such as whether there are any bottlenecks and the quality of your user experience. At the same time, you also need to know how the internals of your application are behaving.

Let’s say you’re running a load test on BlazeMeter in your preproduction environment. You’ll probably run a stress test with increasing amounts of load until it breaks. You finally hit a bottleneck when you get 300 concurrent users on your blog (as shown below):

BlazeMeter has shown you when you will hit the bottleneck. With New Relic, you can get a corresponding set of graphs for this load test in the same time frame to help you uncover where the issues occurred—down to the precise subsystem and resource within that subsystem.

You can also view BlazeMeter’s testing data from within New Relic if you use BlazeMeter’s New Relic plugin.

Testing and Diagnostics in Postproduction

In postproduction, you need to monitor your app or site by simulating the exact scenarios used to test preproduction. It’s crucial to know if something breaks or if the user experience deteriorates and, once again, you need to know how the application’s internals are behaving.

For example: let’s say you’re analyzing a KPI trend graph for a sequence of load tests that you’ve been running automatically, and you notice a sudden spike in the trend. By going over to New Relic’s corresponding charts for the same time period, you can see that it correlates with a spike in the subsystem of the database server and a huge spike of I/O operations on the hard drive resource—which you can then understand was caused by database maintenance undertaken by engineers (as shown below):

KPI Tracking Graph on Loadosophia Showing Sudden Spike in Average Response Time

Continuous automated testing and diagnostics tools are key to ensuring quality at every step of your software life cycle and ultimately essential for a successful and efficient Continuous Delivery process.

Andrey Pokhilko is chief scientist at BlazeMeter. He is a well-known thought-leader and innovator within the Apache JMeter and performance testing community. View posts by Andrey Pokhilko.