Troubleshooting, When F5 Debugging Can’t Help You

in particular, the question is what do you do when this happens?

this situation can crop up in many disguises. for example, you might be seeing a high memory usage in production, or experiencing growing cpu usage over time, or see request times go up, or any of a hundred and one different production issues that make for a hell of a night (somehow, they almost always happen at nighttime)

here is how it usually people think about it.

the first thing to do is to understand what is going on. about the hardest thing to handle in this situations is when we have an issue (high memory, high cpu, etc) and no idea why. usually all the effort is spent just figuring out what and why. . the problem with this process for troubleshooting issues is that it is very easy to jump to conclusions and have an utterly wrong hypothesis. then you have to go through the rest of the steps to realize it isn’t right.

so the first thing that we need to do is gather information. and this post is primarily about the various ways that you can do that. in ravendb, we have actually spent a lot of time exposing information to the outside world, so we’ll have an easier time figuring out what is going on. but i’m going to assume that you don’t have that.

the end all tool for this kind of errors in windbg . this is the low level tool that gives you access to pretty much anything you can want. it is also very archaic and not very friendly at all. the good thing about it is that you can load a dump into it. a dump is a capture of the process state at a particular point in time. it gives you the ability to see the entire memory contents and all the threads. it is an essential tool, but also the last one i want to use, because it is pretty hard to do so. dump files can be very big, multiple gb are very common. that is because they contain the full memory dump of the process. there is also mini dumps, which are easier to work with, but don’t contain the memory dump, so you can watch the threads, but not the data.

the .net memory profiler is another great tool for figuring things out. it isn’t usually so good for production analysis, because it uses the profiler api to figure things out, but it has a wonderful feature of loading dump files (ironically, it can’t handle very large dump files because of memory issues ) and give you a much nicer view of what is going on there.

for high cpu situations, i like to know what is actually going on. and looking at the stack traces is a great way to do that. windbg can help here (take a few mini dumps a few seconds apart), but again, that isn’t so nice to use.

stack dump is a tool that takes a lot of the pain away for having to deal with that. because it just output all the threads information, and we have used that successfully in the past to figure out what is going on.

for general performance stuff “requests are slow”, we need to figure out where the slowness actually is . we have had reports that run the gamut from “things are slow, client machine is loaded” to “things are slow, the network qos settings throttle us”. i like to start by using fiddler to start figuring those things out. in particular, the statistics window is very helpful:

the obvious things are the bytes sent & bytes received. we have a few cases where a customer was actually sending 100s of mb in either of both directions, and was surprised it took some time. if those values are fine, you want to look at the actual performance listing. in particular, look at things like tcp/ip connect, time from client sending the request to server starting to get it, etc.

if you found the problem is actually at the network layer, you might not be able to immediately handle it. you might need to go a level or two lower, and look at the actual tcp traffic. this is where something like wire shark comes into play, and it is useful to figure out if you have specific errors at  that level (for example, a bad connection that cause a lot of packet loss will impact performance, but things will still work).

other tools that are very important include resource monitor , process explorer and process monitor . those give you a lot of information about what your application is actually doing .

one you have all of that information, you can form a hypothesis and try to test it.

if you own the application in question, the best way to improve your chances of figuring out what is going on is to add logging. lots & lots of logging. in production, having the logs to support what is going on is crucial. i usually have several levels of logging. for example, what is the traffic in/out of my system. next there is the actual system operations, especially anything that happens in the background. finally, there are the debug/trace endpoints that will expose internal state and allow you to tweak various things at runtime.

having good working knowledge on how to properly utilize the above mention tools is very important, and should be considered to be much more useful than learning a new api or a language feature.