I encountered an interesting gotcha today, which I thought would be interesting to share with you. The symptoms were the following: I wrote a simple C++ function exported from a DLL, and tried to invoke it using a P/Invoke wrapper. The C++ function returns a bool, and was declared as follows (the code was simplified for expository purposes):

extern "C" __declspec(dllexport) bool IsPrime(int n) { 
    if (n <= 1) return false; 
    if (n == 2) return true; 
    if (n % 2 == 0) return false; 
    for (int i = 3; i < n; ++i) { 
        if (n % i == 0) return false; 
    } 
    return true; 
}

The managed counterpart was:

[DllImport(@"..\..\..\Debug\NativeDll.dll", 
           CallingConvention=CallingConvention.Cdecl)] 
static extern bool IsPrime(int n);

At runtime, for some values of n, the output would be inconsistent. Specifically, when n=0 or n=1, the managed wrapper would return true – although the C++ function clearly returns false. For other values, however, the wrapper returned the right values.

Next, I disassembled the C++ function, to find the following code responsible for the first branch:

cmp dword ptr[n], 1 
jg BranchNotTaken 
xor al, al 
jmp BailOut 
… 
BailOut: 
… 
ret

In other words, to return false, the function clears out the AL register (which is the lowest byte of the EAX register), and then returns. What of the rest of the EAX register? In the debug build, it’s likely to contain 0xCCCCCCCC, because of this part of the function’s prologue:

lea edi,[ebp-0CCh]  
mov ecx,33h  
mov eax,0CCCCCCCCh  
rep stos dword ptr es:[edi]

To conclude, the function returns 0xCCCCCC00 in the EAX register, which is then interpreted as true by the managed wrapper. Why? Because by default, P/Invoke considers bool parameters to be four-byte values, akin to the Win32 BOOL, and then any non-zero value is mapped to true.

To fix this, I had to convince P/Invoke to marshal back the return value as a single byte. There’s no dedicated type for that in the UnmanagedType enumeration, but U1 (unsigned byte) or I1 (signed byte) will do the trick:

[DllImport(@"..\..\..\Debug\NativeDll.dll", 
           CallingConvention=CallingConvention.Cdecl)] 
[return: MarshalAs(UnmanagedType.U1)] 
static extern bool IsPrime(int n);