Debugging in Production
Debugging in Production
Debugging in production isn't easy. There's a better and safer way to find bugs that are affecting your users: Application Performance Monitoring.
Join the DZone community and get the full member experience.Join For Free
Sensu is an open source monitoring event pipeline. Try it today.
Why Debugging in Production Is So Tempting
In one of my first jobs, one of the tasks I had was to fix a bug which used to occur from time to time in a very complex production system. "That is easy!" I thought. "I will reproduce the same situation in my development environment, find the broken line, implement a quick fix, and it’s done!" However, it turned out that reproducing exactly the same scenario which occurred in production was not possible, so after some time, I gave up this idea. I was forced to spend days analyzing logs and trying to correlate many different events to come up with an idea of what might have happened.
Soon, I realized that it’s as tedious as looking for a needle in a haystack. A couple of fruitless days later, I came to the conclusion that I would need to add more logging here and there and wait a couple of days or even months to see if the bug occurred again. Then I thought that hunting bugs in production is somehow crude compared to the sophisticated tools we have when developing an application. You’re implementing a new feature and seeing that the result of what your service returned is not what you had expected? You just put a few breakpoints and click the Debug button! A few moments later, you know exactly what happened. Wouldn’t it be awesome to do the same in a production environment?
Why Debugging in Production Is So Hard
"Wait a second!" you might have thought. "Don’t we have the remote debugging features in most of the modern IDEs? Couldn’t we just connect to the running production application and debug it as we do from our local environment?" While it’s possible, another problem arises: most of our business applications handle many requests per second. There is no easy way to control breakpoints firing everywhere when your application is being remotely debugged. As you can imagine, we don’t want to block all of our users from using our application when we decided to debug it. More often than not, we also can’t just force our application to reproduce the bug which happened yesterday; sometimes the only way to do it is to wait until it happens again to one of our users. Thus, keeping a remote debug session in production without a strict control of how breakpoints fire is like putting landmines in the forest and inviting our users to run through it.
A Better, and Above All, Safer Way
FusionReactor is an Application Performance Monitor, which comes with many advanced capabilities which you wouldn’t normally expect to find in a monitoring solution. One of these is the production debugger, designed to allow you to get low-level debug information from your production runtime environment.
One of the main issues you would be faced with using some of the traditional debuggers is that once a breakpoint is set, it will fire for any thread which crosses that point in the code. FusionReactor overcomes this by having a range of techniques to control the way a breakpoint should fire. For example, it can limit the number of times (threads) that a given breakpoint will trigger, which solves the problem of impacting too many users. Need more ways to control it? You can even configure a breakpoint to fire for a user from a specific IP address (session), when a specific variable matches a value, or when a specific exception takes place. However, what if a breakpoint triggers at night when nobody from your team is watching? FusionReactor allows you to define thread pause timeouts so if you don't intercept a paused thread within a specific time, the debugger will release the lock and allow thread execution to continue. When used with the thread limits, this reduces the possible impact to one thread only, and only for n seconds.
Another benefit is that FusionReactor can send out an email with the stack-trace and variables at the point that the trigger fires. This gives you a very flexible and unobtrusive way to be notified with plenty of information to make debugging easier than ever before.
Debugging in production doesn’t have to be cumbersome. FusionReactor is shipped with a fully integrated IDE-style debugger which runs directly in your browser — no need to install additional fat clients to start remote debugging. Everything is built-in and ready to go.
Published at DZone with permission of Grzegorz Mirek , DZone MVB. See the original article here.
Opinions expressed by DZone contributors are their own.