Over a million developers have joined DZone.

Profiling .NET Performance Issues

DZone's Guide to

Profiling .NET Performance Issues

We ended up with many garbage collection operations because of some byte arrays — and that resulted in a performance hit.

· Performance Zone ·
Free Resource

Maintain Application Performance with real-time monitoring and instrumentation for any application. Learn More!

In this post, I want to talk about a frustrating problem most developers will encounter during their career: performance issues.

You write some code, you test it, you run it locally, and it works fine — but once it is deployed, bad things start to happen.

It simply refuses to give you the performance that you expect to.

Besides doing the obvious (calling the server some bad names), what else can you do?

In the latest use case that we encountered, one of our engineers was working on merging the code from a few processes into a single process. We expected the performance to stay the same or even to improve (no need for inter-process communication). And in all of the local tests, it did.

However, when deployed to production, things started getting weird.

At first, the performance was great. But then, it started deteriorating for no apparent reason.

CPU started to spike and the total throughput went down to about 25% worse than the original throughput.  

The engineer who was the assigned to investigate the issue started by digging into the process performance indicators using ELK.

Now, we are talking about a deployment of multiple processes per server and of multiple servers — so, careful consideration should go into aggregating the data.

Here is a sample of some interesting charts:

Image title

Analyzing the results, we realized the problem happened on all of the servers intermittently.

We also realized that some inputs will cause the problem to be more serious than others.

We used the Ants profiling tool on a single process and fed it with some problematic inputs. The results were, surprisingly, not very promising:

  • There were no unexpected hotspots.

  • There were no memory leaks.

  • Generation 2 collection was not huge, but it had a lot of data — more than Gen1 (but less than Gen0).

Well, this got us thinking: might our problem be GC-related?

We now turned to the Perfmon tool.

Analyzing %time in the GC metric revealed that some processes spent as much as 50% of their time doing GC.

Profiling .NET performance issues

Now, the chips started falling.

One of our original processes used to do some bookkeeping, holding some data in memory for a long duration. Another type of a process was a typical worker: doing a lot of calculations using some byte arrays and then quickly dumping them.

When the two processes were merged, we ended up with a lot of data in Gen2. We also ended up with many garbage collection operations because of the byte arrays — and that resulted in a performance hit.

Well, once we knew what was the problem, we had to resolve it. But this is an entirely different blog post altogether...

Collect, analyze, and visualize performance data from mobile to mainframe with AutoPilot APM. Get a Demo!

.net ,profiling ,performance ,garbage collection

Published at DZone with permission of

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}