I spent some time the last couple of days working on some C# code that used P/Invoke to interact with an unmanaged library consisting of several function calls and some callback procedures.
One of the issues I ran into was that I needed to pass to one of the library's functions a delegate for it to use as a callback procedure. Doing this is usually not much of a problem, but this time I'd get an AccessViolationException after my managed callback function returned, signaling that the stack was probably getting corrupted somewhere.
After looking around quite a bit I realized my mistake: The C API I was invoking used the C calling convention (__cdecl), instead of the __stdcall convention used in the Windows API.
I was correctly decorating my P/Invoke signatures with CallingConvention=CallingConvention.Cdecl in the [DllImport] attribute, but this only affect the APIs I called directly; the delegate instance I was passing as callback was getting marshaled as a __stdcall procedure, thus causing the problem.
I remembered that in the old days of .NET 1.0/1.1 there was nothing built into the .NET framework that you could directly use to signal the P/Invoke layer to marshal a delegate as a __cdecl function, so you usually had to resort to creating a little wrapper using Managed C++ or, even runtime code generation.
Fortunately, I discovered that .NET 2.0 had built-in support for this, though finding it took a while. Hopefully this post will make it easier to find in the future.
[UnmanagedFunctionPointer]
The .NET Framework 2.0 contains a the new UnmanagedFunctionPointerAttribute class, which can be used to decorate a delegate declaration. One of the fields of this attribute can be used to specify the calling convention you want to use when an instance of the delegate needs to be marshaled to unmanaged code as a callback function.
With this, I could simply declare my delegate like this:
delegate void CallbackProc(IntPtr clientData);
Nifty and very useful stuff.
