DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports Events Over 2 million developers have joined DZone. Join Today! Thanks for visiting DZone today,
Edit Profile Manage Email Subscriptions Moderation Admin Console How to Post to DZone Article Submission Guidelines
View Profile
Sign Out
Refcards
Trend Reports
Events
Zones
Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Partner Zones AWS Cloud
by AWS Developer Relations
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Partner Zones
AWS Cloud
by AWS Developer Relations
Building Scalable Real-Time Apps with AstraDB and Vaadin
Register Now

Trending

  • Clear Details on Java Collection ‘Clear()’ API
  • Guide To Selecting the Right GitOps Tool - Argo CD or Flux CD
  • Integration Testing Tutorial: A Comprehensive Guide With Examples And Best Practices
  • Five Java Books Beginners and Professionals Should Read

Trending

  • Clear Details on Java Collection ‘Clear()’ API
  • Guide To Selecting the Right GitOps Tool - Argo CD or Flux CD
  • Integration Testing Tutorial: A Comprehensive Guide With Examples And Best Practices
  • Five Java Books Beginners and Professionals Should Read

Troubleshooting, When F5 Debugging Can’t Help You

Oren Eini user avatar by
Oren Eini
·
Sep. 08, 14 · Interview
Like (0)
Save
Tweet
Share
3.93K Views

Join the DZone community and get the full member experience.

Join For Free

you might have noticed that we have been doing a lot of work on the operational side of things. to make sure that we give you as good a story as possible with regards to the care & feeding of ravendb. this post isn’t about this. this post is about your applications and systems, and how you are going to react when !@)(*#!@(* happens.

in particular, the question is what do you do when this happens?

this situation can crop up in many disguises. for example, you might be seeing a high memory usage in production, or experiencing growing cpu usage over time, or see request times go up, or any of a hundred and one different production issues that make for a hell of a night (somehow, they almost always happen at nighttime)

here is how it usually people think about it.

the first thing to do is to understand what is going on. about the hardest thing to handle in this situations is when we have an issue (high memory, high cpu, etc) and no idea why. usually all the effort is spent just figuring out what and why. . the problem with this process for troubleshooting issues is that it is very easy to jump to conclusions and have an utterly wrong hypothesis. then you have to go through the rest of the steps to realize it isn’t right.

so the first thing that we need to do is gather information. and this post is primarily about the various ways that you can do that. in ravendb, we have actually spent a lot of time exposing information to the outside world, so we’ll have an easier time figuring out what is going on. but i’m going to assume that you don’t have that.

the end all tool for this kind of errors in windbg . this is the low level tool that gives you access to pretty much anything you can want. it is also very archaic and not very friendly at all. the good thing about it is that you can load a dump into it. a dump is a capture of the process state at a particular point in time. it gives you the ability to see the entire memory contents and all the threads. it is an essential tool, but also the last one i want to use, because it is pretty hard to do so. dump files can be very big, multiple gb are very common. that is because they contain the full memory dump of the process. there is also mini dumps, which are easier to work with, but don’t contain the memory dump, so you can watch the threads, but not the data.

the .net memory profiler is another great tool for figuring things out. it isn’t usually so good for production analysis, because it uses the profiler api to figure things out, but it has a wonderful feature of loading dump files (ironically, it can’t handle very large dump files because of memory issues smile ) and give you a much nicer view of what is going on there.

for high cpu situations, i like to know what is actually going on. and looking at the stack traces is a great way to do that. windbg can help here (take a few mini dumps a few seconds apart), but again, that isn’t so nice to use.

stack dump is a tool that takes a lot of the pain away for having to deal with that. because it just output all the threads information, and we have used that successfully in the past to figure out what is going on.

for general performance stuff “requests are slow”, we need to figure out where the slowness actually is . we have had reports that run the gamut from “things are slow, client machine is loaded” to “things are slow, the network qos settings throttle us”. i like to start by using fiddler to start figuring those things out. in particular, the statistics window is very helpful:

image

the obvious things are the bytes sent & bytes received. we have a few cases where a customer was actually sending 100s of mb in either of both directions, and was surprised it took some time. if those values are fine, you want to look at the actual performance listing. in particular, look at things like tcp/ip connect, time from client sending the request to server starting to get it, etc.

if you found the problem is actually at the network layer, you might not be able to immediately handle it. you might need to go a level or two lower, and look at the actual tcp traffic. this is where something like wire shark comes into play, and it is useful to figure out if you have specific errors at  that level (for example, a bad connection that cause a lot of packet loss will impact performance, but things will still work).

other tools that are very important include resource monitor , process explorer and process monitor . those give you a lot of information about what your application is actually doing .

one you have all of that information, you can form a hypothesis and try to test it.

if you own the application in question, the best way to improve your chances of figuring out what is going on is to add logging. lots & lots of logging. in production, having the logs to support what is going on is crucial. i usually have several levels of logging. for example, what is the traffic in/out of my system. next there is the actual system operations, especially anything that happens in the background. finally, there are the debug/trace endpoints that will expose internal state and allow you to tweak various things at runtime.

having good working knowledge on how to properly utilize the above mention tools is very important, and should be considered to be much more useful than learning a new api or a language feature.

Dump (program)

Published at DZone with permission of Oren Eini, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

Trending

  • Clear Details on Java Collection ‘Clear()’ API
  • Guide To Selecting the Right GitOps Tool - Argo CD or Flux CD
  • Integration Testing Tutorial: A Comprehensive Guide With Examples And Best Practices
  • Five Java Books Beginners and Professionals Should Read

Comments

Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com

Let's be friends: