Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

Tips for Working With Native Memory

DZone's Guide to

Tips for Working With Native Memory

What this code does is pretty simple...and brutal. It allocates memory in a way that absolutely guarantees that you can’t get away with a whole host of memory problems.

· Performance Zone
Free Resource

Transform incident management with machine learning and analytics to help you maintain optimal performance and availability while keeping pace with the growing demands of digital business with this eBook, brought to you in partnership with BMC.

I want to start by saying that this isn’t my idea; I read about it a few times, and I recently encountered it with sodium_malloc, so I decided to write my own implementation of the world’s most expensive memory allocator.

public static unsafe class ElectricFencedMemory
{
[DllImport("kernel32.dll", SetLastError = true)]
private static extern byte* VirtualAlloc(byte* lpAddress, UIntPtr dwSize,
AllocationType flAllocationType, MemoryProtection flProtect);
​
[DllImport("kernel32.dll", SetLastError = true)]
private static extern bool VirtualProtect(byte* lpAddress, UIntPtr dwSize,
MemoryProtection flNewProtect, out MemoryProtection lpflOldProtect);
​
[DllImport("kernel32.dll", SetLastError = true)]
private static extern bool VirtualFree(byte* lpAddress, UIntPtr dwSize,
FreeType dwFreeType);
​
[DllImport("msvcrt.dll", EntryPoint = "memset", CallingConvention = CallingConvention.Cdecl, SetLastError = false)]
[SecurityCritical]
private static extern IntPtr memset(byte* dest, int c, long count);
​
​
[Flags]
private enum AllocationType : uint
{
COMMIT = 0x1000,
}
​
[Flags]
private enum MemoryProtection : uint
{
NOACCESS = 0x01,
READWRITE = 0x04,
}
​
[Flags]
private enum FreeType : uint
{
MEM_DECOMMIT = 0x4000,
}
​
public static byte* Allocate(int size)
{
var remaining = size % 4096;
var sizeInPages = (size / 4096) + (remaining == 0 ? 0 : 1);
​
var allocatedSize = ((sizeInPages + 2) * 4096);
var virtualAlloc = VirtualAlloc(null, (UIntPtr)allocatedSize, AllocationType.COMMIT,
MemoryProtection.READWRITE);
if (virtualAlloc == null)
throw new Win32Exception();
​
*(int*)virtualAlloc = allocatedSize;
​
MemoryProtection protect;
if (VirtualProtect(virtualAlloc, (UIntPtr)(4096), MemoryProtection.NOACCESS,
out protect) == false)
throw new Win32Exception();
​
if (VirtualProtect(virtualAlloc + (sizeInPages + 1) * 4096, (UIntPtr)(4096), MemoryProtection.NOACCESS,
out protect) == false)
throw new Win32Exception();
​
var firstWritablePage = virtualAlloc + 4096;
​
memset(firstWritablePage, 0xED, 4096 * sizeInPages); // don't assume zero'ed mem
if (remaining == 0)
return firstWritablePage;
// give the memory out so its end would be at the 2nd guard page
return firstWritablePage + (4096 - remaining);
​
}
​
public static void Free(byte* p)
{
var remaining = (int)((long)p % 4096);
var firstWritablePage = p - remaining;
for (int i = 0; i < remaining; i++)
{
if (firstWritablePage[i] != 0xED)
throw new InvalidOperationException("Invalid memory usage, you killed Ed!");
}
MemoryProtection protect;
var address = firstWritablePage - 4096;
// this will access the memory, which will error if this was already freed
if (VirtualProtect(address, (UIntPtr)4096, MemoryProtection.READWRITE, out protect) ==
false)
throw new Win32Exception();
var dwSize = *(int*)address;
​
// decommit, not release, they are not available for reuse again, any
// future access will throw
if (VirtualFree(address, (UIntPtr)dwSize, FreeType.MEM_DECOMMIT) == false)
throw new Win32Exception();
}
}

What this code does is pretty simple, and quite brutal. It allocates memory in such a fashion that absolutely guarantees that you can’t get away with a whole host of memory problems.

For example, if you try to overwrite a buffer allocated by this method, you’ll immediately hit the guard page and die horribly (and predictably, in the place where the error actually happened, not a long way off). If you somehow write before the buffer, that will be detected on free if it's a small underwrite (which tend to be much rarer, by the way), or immediately if it's a big change.

What's more, once the memory is freed, it is poisoned and can never be used again. This pretty much relies on us running on 64 bits with effectively unlimited virtual memory and has the nasty side effect of turning a 50 bytes allocation to something requiring 12 KB. Having said that, as a debugging tool, this is invaluable.

And yes, I’m aware that windows already has that with the heap verifier. However, as I’m using this in.NET code, I needed to write my own (this also pretty much works the same way with Linux; you just need to switch the API, but the functionality is the same).

This was built because we were chasing a memory corruption error. I ran this, but it pointed me to a totally different location than suspected. So, it is either doing a very good job or it found me another issue.

I'm investigating…

Or, as the case may be, we found a bug in the actual memory guard (we didn’t handle allocations of exact page size correctly, and they broke), but at least it broke consistently and was pretty easy to find once I looked in the right place!

Evolve your approach to Application Performance Monitoring by adopting five best practices that are outlined and explored in this e-book, brought to you in partnership with BMC.

Topics:
performance ,native memory ,bugs ,memory guard

Published at DZone with permission of Oren Eini, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}