P/Invoke with C++ bool Return Values
Join the DZone community and get the full member experience.
Join For Free
i encountered an interesting gotcha today, which i thought would be interesting to share with you. the symptoms were the following: i wrote a simple c++ function exported from a dll, and tried to invoke it using a p/invoke wrapper. the c++ function returns a
bool
, and was declared as follows (the code was simplified for expository purposes):
extern "c" __declspec(dllexport) bool isprime(int n) { if (n <= 1) return false; if (n == 2) return true; if (n % 2 == 0) return false; for (int i = 3; i < n; ++i) { if (n % i == 0) return false; } return true; }
the managed counterpart was:
[dllimport(@"..\..\..\debug\nativedll.dll", callingconvention=callingconvention.cdecl)] static extern bool isprime(int n);
at runtime, for some values of n , the output would be inconsistent. specifically, when n =0 or n =1, the managed wrapper would return true – although the c++ function clearly returns false . for other values, however, the wrapper returned the right values.
next, i disassembled the c++ function, to find the following code responsible for the first branch:
cmp dword ptr[n], 1 jg branchnottaken xor al, al jmp bailout … bailout: … ret
in other words, to return
false
, the function clears out the al register (which is the lowest byte of the eax register), and then returns. what of the rest of the eax register? in the debug build, it’s likely to contain 0xcccccccc, because of this part of the function’s prologue:
lea edi,[ebp-0cch] mov ecx,33h mov eax,0cccccccch rep stos dword ptr es:[edi]
to conclude, the function returns 0xcccccc00 in the eax register, which is then interpreted as true by the managed wrapper. why? because by default, p/invoke considers bool parameters to be four-byte values, akin to the win32 bool , and then any non-zero value is mapped to true .
to fix this, i had to convince p/invoke to marshal back the return value as a single byte. there’s no dedicated type for that in the unmanagedtype enumeration, but u1 (unsigned byte) or i1 (signed byte) will do the trick:
[dllimport(@"..\..\..\debug\nativedll.dll", callingconvention=callingconvention.cdecl)] [return: marshalas(unmanagedtype.u1)] static extern bool isprime(int n);
i am posting short links and updates on twitter as well as on this blog. you can follow me: @goldshtn
Published at DZone with permission of Sasha Goldshtein, DZone MVB. See the original article here.
Opinions expressed by DZone contributors are their own.
Comments