Announcing Tracer: A Generic Way to Track Resource Usage and Leaks
Tracer is a WinDbg extension I wrote last month to diagnose a resource leak that is not covered by well-known facilities like
!htrace or UMDH. Tracking any resource leak starts with understanding where you are acquiring the resource and neglecting to release it – and with Tracer, you can do this for any kind of resource.
The basic process of hunting for resource leaks is quite simple. For example, consider what UMDH does on your behalf. UMDH enables support in the operating system (specifically, in the Heap Manager that resides in ntdll.dll) to capture stack traces of heap memory allocations and correlate the allocation requests with deallocation (free) requests. If the application exhibits a pattern of leaking memory, you will see specific call stacks reported as responsible for memory allocations that have not been freed. And – even though you don’t know for sure who was supposed to free that memory – you are given a good clue in the form of the piece of code that performed the memory allocation.
With Tracer, you can perform this process automatically for any kind of resource. Tracer exposes two extension commands for tracking resource usage –
!traceclose, and an additional extension command for displaying reports of resource leaks –
!tracedisplay. You typically call the tracking commands from breakpoints strategically positioned where you acquire and release the resource you’re tracking.
Let’s take a look at an example. Suppose that you’re tracking a database connection leak. You create database connections in the db_connect function, and close them in the db_close function. To track a database connection leak, place breakpoints in these functions and call the tracking commands from these breakpoints, as follows:
bp dbmodule!db_connect "gu; !traceopen @eax; gc" bp dbmodule!db_close "!traceclose poi(@esp+4); gc"
The preceding commands assume that the
db_connect function returns the database connection handle, and that the
db_close function takes the database connection handle as a parameter on the stack (if you have private symbols for
db_close, you can of course reference the parameter name instead of relying on its stack location).
Now you can run your application and wait for the suspicious leak to accumulate. Whenever you'd like, you can break into the debugger and inspect the current status of the specific resource you're tracking using the
!tracedisplay -stats command. This will dump out statistics (in sorted order) for the call stacks acquiring database connections without subsequently closing them. Here is a fictituous example:
0:000> !tracedisplay -stats Total objects with events : 0n34 Total events with stack traces : 0n34 ----- STACK #1 OPEN=0n30 CLOSE=0n0 OTHER=0n0 WEIGHT=0n30 ----- MyApp!OpenDatabaseConnection+0x3c MyApp!LogWriterThread+0x4b KERNEL32!BaseThreadInitThunk+0xe ntdll!__RtlUserThreadStart+0x72 ntdll!_RtlUserThreadStart+0x1b ----- STACK #2 OPEN=0n4 CLOSE=0n0 OTHER=0n0 WEIGHT=0n4 ----- MyApp!OpenDatabaseConnection+0x3c MyApp!InitializationThread+0x60 KERNEL32!BaseThreadInitThunk+0xe ntdll!__RtlUserThreadStart+0x72 ntdll!_RtlUserThreadStart+0x1b
Often, the resources you acquire and release have weights associated with them - not every instance of the resource will have the same cost. This obviously applies to memory (large allocation leaks are of more concern than small ones), but might also apply to other kinds of resources. To tell Tracer that your resources have weights, use the
!traceopen <object> <weight> command. For example, when tracing memory leaks through RtlAllocateHeap, you can use the following command (on x86):
bp ntdll!RtlAllocateHeap "r $t0 = poi(@esp+0n12); gu; !traceopen @eax @$t0; gc"
As a result, Tracer will display object weights and will sort the statistics output by weights so you can easily identify call stacks responsible for large resource leaks and fix them first.
It’s worth mentioning that Tracer can be used to track invalid resource usage as well as resource leaks. For example, you could use Tracer to immediately know when you are releasing a resource that was not previously acquired by your code. If that happens, Tracer will output a diagnostic message to the WinDbg console. Also, you can use
!traceclose -keepstack <object> to indicate that you want to retain the close call stack even if the object's allocation-deallocation delta is now 0. This helps understand where the object was created and destroyed after the fact. Just use the
!tracedisplay <object> command to view information about that particular object.
A final note about performance is due here. As you can see, Tracer relies on breakpoints that call the tracking extension commands. This is - without doubt - the most expensive part of the tracking mechanism. For memory allocation tracking in large applications, UMDH is a much better idea because it does not require a debugger and does not rely on breakpoints. Off the top of my head I'd say that if you're acquiring and releasing resources more than 100 times per second, you will probably see a considerable slowdown when tracking them with Tracer. For many kinds of resources, though, this is a reasonable limit - I have used Tracer with file handles, sockets, database connections, and database transactions.
I am posting short links and updates on Twitter as well as on this blog. You can follow me:@goldshtn