We Keep Coding

C# Pinvoke IntPtr in structs and memory

If i have understood right, when using structs with non blitable values, the struct data from unmanaged memory is copied into the managed memory (basically having the same struct twice).
Also if im not wrong, IntPtr variables, are stored in managed memory, but the data they point to is in the unmanaged memory.
Now lets say i have a delegate method, which is being called by a c++ function and receives a struct as ref, and a handler would be something like:
private void _handler(ref MyStruct p){}
Now the api says I should not keep the reference to the struct as the memory might be recycled and used for further calls, but since the unmanaged struct is copied into managed memory, and after I assign it to a local variable it is copied again (because it is a struct!) i shouldnt have any issues at all if the unmanaged memory gets freed or re written.
However, if the struct contains an IntPtr variable, i might save a copy of the pointer but not of the data it points, so if i try to access the IntPtr data from my saved struct, i might get some memory fault, is this correct?
One last question is, doing changes to the struct in managed memory, will also affect the unmanaged memory struct, since it is passed by ref, and it implicitly means IN/OUT, so after the event handler call ends, changes in managed memory will be made in the unmanaged memory as well?

Your understanding is largely correct. Taking away the details of blittabiltiy, the root question is one of ownership of the memory being passed to your callback. Typically, the contract between the caller of the callback and the callback itself is that the memory is owned by the caller and will be valid only for the duration of the callback itself. (Other contracts are possible as well - for instance, the caller could transfer ownership of the memory to the callback, in which case the callback is responsible for freeing the memory when it is done).
In the documentation you reference, the API is telling you not to keep a reference to the memory, which means that you are very likely in the standard case where the caller owns the structure. What this means is that both the contents of the structure and everything pointed to by that structure should be valid during the call to your callback but cannot be relied upon after your callback completes.
Copying a structure to a local wouldn't be problematic because the native memory should not be freed until after your callback completes, so keeping as many copies as you need during the execution of the callback should be fine. What would be problematic is saving a copy of the structure to a location, such as a static field, which will be alive after your callback returns.
Since the structure is implicitly in/out, then modifications you make to it will be reflected on the native side of the call as well. This is true regardless of if the structure contained only blittable data. If the structure contained non-blittable data then the CLR would marshal your changes to the structure back out to the native copy when your code completes.
Note that just because you can modify the native structure does not mean that the API contract expects you to do so. Frequently API signatures contain pointers to structures in C++ in order to avoid copying the entire structure when making a call, not to enable modification of the structure.

Convention for passing BSTRs into COM functions from C# (COM interop)

I am writing writing an API in COM in C++, and also writing a program which consumes this API in C#. My question is about BSTR memory management semantics when passing BSTRs into COM functions. Say my IDL looks like:
HRESULT SomeFunction([in] BSTR input);
Currently this function is implemented like this:
HRESULT SomeFunction(BSTR input) {
// Do stuff ..., then:
SysFreeString(input);
}
When I call it from C# with something like SomeFunction(myString), will C# generate something like this (pseudocode):
myString = SysAllocString("string");
SomeFunction(myString);
Or rather like this:
myString = SysAllocString("string");
SomeFunction(myString);
SysFreeString(myString);
That is, does C# free the BSTR that it generates to marshal to the COM interface, or should I free it inside my function? Thanks!

From Allocating and Releasing Memory for a BSTR:
When you call into a function that expects a BSTR argument, you
must allocate the memory for the BSTR before the call and
release it afterwards. ...
So don't free it if it is an input parameter. C# (and any other runtime that uses COM objects) must respect the COM convention for managing memory pass in and out of COM objects, and must therefore manage the memory for the string if it is an input parameter. Otherwise, how would a COM object know that it is being called from C# or some other language runtime?
Additional google-fu turned up this: Marshaling between Managed and Unmanaged Code
... Regarding ownership issues, the CLR follows COM-style
conventions:
Memory passed as [in] is owned by the caller and should be both
allocated by the caller and freed by the caller. The callee should
not try to free or modify that memory.
Memory allocated by the callee and passed as [out] or returned
is owned by the caller and should be freed by the caller.
The callee can free memory passed as [in, out] from the caller,
allocate new memory for it, and overwrite the old pointer value,
thereby passing it out. The new memory is owned by the caller. This
requires two levels of indirection, such as char **.
In the interop world, caller/callee becomes CLR/native code. The rules
above imply that in the unpinned case, if when in native code you
receive a pointer to a block of memory passed to you as [out] from
the CLR, you need to free it. On the other hand, if the CLR receives
a pointer that is passed as [out] from native code, the CLR needs to
free it. Clearly, in the first case, native code needs to do the
de-allocation and in the second case, managed code needs to do
de-allocation.
So the CLR follows the COM rules for memory ownership. QED.

Do you mean from the point of view of the C# developer or from the C++ developer.
The C# developer should not have to worry about any memory management when dealing with COM+.
Creating a COM+ component in C++, you would not have to know who is calling you, the memory semantics are the same. If it's an in parameter, the caller is responsible for managing the memory, regardless of whether it's C++ or C#. In C#, the CLR takes care of it for them.

Memory Management Of Unmanaged Component By CLR

I am having a little confusion , may be this question is very silly one.
where does the memory allocated for a unmanaged component?
In my .net code if i initiated an unmanaged component, where this component is going to be loaded and memory is allocated ?
How CLR marshall call between Managed and Unmanaged heap ?
EDIT
Thanks for your reply but what i am asking is say suppose i do a DLLIMPORT of User32.Dll , this is clearly a unmanaged dll and i call some function in User32.DLL now my question , how CLR marshall my call to this unmanged dll?

It starts out pretty easy. The pinvoke marshaller first calls LoadLibrary and passes the DLL name you specified, the DllImportAttribute.Value property. In your case, user32.dll is already loaded because it gets loaded by the .NET bootstrapper, its reference count just gets incremented. But normally the Windows loader gets the DLL mapped into the address space of the process so the exported functions can be called.
Next is GetProcAddress to get the address of the function to call, the DllImportAttribute.EntryPoint property. The marshaller makes a couple of tries unless you used ExactSpelling. A function name like "foo" is tested several possible ways, foo and fooW or fooA. Nasty implementation detail of Win32 related to the difference between Unicode and Ansi characters. The CharSet property matters here.
Now I need to wave hands a bit because it gets tricky. The marshaller constructs a stack frame, setting up the arguments that need to be passed to the exported function. This requires low level code, carefully excluded from prying eyes. Take it at face value that it performs the kind of translations that the Marshal class supports to convert between managed and unmanaged types. The DllImportAttribute.CallingConvention property matters here because that determines what argument value needs to be place where so that the called function can read it properly.
Next it sets up an SEH exception handler so that hardware exceptions raised by the called code can be caught and translated into a managed exception. The one that generates the more common one, AccessViolationException. And others.
Next, it pushes a special cookie on the stack to indicate that unmanaged code is about to start using stack. This prevents the garbage collector from blundering into unmanaged stack frames and interpret the pointers it finds there as managed object references. You can see this cookie back in the debugger's call stack, [Managed to Native Transition].
Next, just an indirect call to the function address as found with GetProcAddress(). That gets the unmanaged code running.
After the call, cleanup might need to be done to release memory that was allocated to pass the unmanaged arguments. The return value might need to be translated back to a managed value. And that's it, assuming nothing nasty happened, execution continues on the next managed code statement.

Unmanaged memory allocations come from the process heap. You are responsible for allocating/deallocating the memory, since it will not get garbage collected because the GC does not know about these objects.

Just as an academic piece of info expanding on what has been posted here:
There are about 8 different heaps that the CLR uses:
Loader Heap: contains CLR structures and the type system
High Frequency Heap: statics, MethodTables, FieldDescs, interface map
Low Frequency Heap: EEClass, ClassLoader and lookup tables
Stub Heap: stubs for CAS, COM wrappers, P/Invoke
Large Object Heap: memory allocations that require more than 85k bytes
GC Heap: user allocated heap memory private to the app
JIT Code Heap: memory allocated by mscoreee (Execution Engine) and the JIT compiler for managed code
Process/Base Heap: interop/unmanaged allocations, native memory, etc
HTH

Part of your question is answered by Michael. I answer the other part.
If CLR loaded into an unmanaged process, it is called CLR hosting. This usually involves calling an entry point in mscoree DLL and then the default AppDomain is loaded. In such a case, CLR asks for a block of memory from the process and when given, that becomes its memory space and will have a stack and heap.

C# deallocate memory referenced by IntPtr

I am using some unmanaged code that is returning pointers (IntPtr) to large image objects. I use the references but after I am finished with the images, I need to free that memory referenced by the pointers. Currently, the only thing that frees the memory is to shutdown my entire app. I need to be able to free that memory from inside my application.
Here is the call to that allocates the memory. hbitmap is the pointer that is returned and needs to be deallocated.
[DllImport("twain_32.dll", EntryPoint = "#1")]
public static extern TwainResult DsImageTransfer(
[In, Out] Identity origin, [In] Identity dest, DataGroup dg,
DataArgumentType dat, Message msg, ref IntPtr hbitmap);

You need to use the specific memory allocator mechanism that was used to allocate the memory in the first place.
So, if you were using COM and the IMalloc interface to allocate the memory, then you have to pass the IntPtr back to the Free method on that implementation in order to free the memory allocated.
If you are indeed using the COM allocator that is returned by a call to CoGetMalloc, then you can call the static FreeCoTaskMem method on the Marshal class.
The Marshal class also has a method for freeing memory that is allocated through a call to LocalAlloc called FreeHGlobal.
However, and this is a common case, if the memory was allocated by the new operator in C++, or a call to malloc in C, then you have to expose a function in unmanaged code through interop which will free the memory appropriately.
In the case of C++, you would expose a function that takes a pointer and simply calls delete on that pointer. In the case of malloc, you would create a function that takes a pointer, and calls free on that pointer.
In specific regards to your question, it would seem that DsImageTransfer is a vendor-specific API (one that doesn't have much discoverability on the web either, I'm afraid), so more information is needed about that specific API function and how it allocates memory. Just knowing the handle type (an HBITMAP in this case) doesn't give any indication as to how it's allocated. It can be allocated with all the mechanisms mentioned above.
Assuming it is creating the HBITMAP using GDI Object api functions (specifically, the CreateBitmap function), then you could use the DeleteObject function to release the handle (as per the documentation page for the GDI Object API functions).

That would depend on how that memory was allocated. The Marshal class has methods for deallocating memory allocated through the common interop allocation patterns, like FreeCoTaskMem. If the unmanaged code uses a non Interop compatible way of allocating, then you cannot interop with it.
Updated
If I would venture a guess, the function #1 you invoke in twain_32.dll is the DS_ENTRY function in a TWAIN provider. The Twain specifications call out the memory resource management protocol:
Memory Management in TWAIN 2.0 and
Higher
TWAIN requires Applications and
Sources to manage each other’s memory.
The chief problem is guaranteeing
agreement on the API’s to use. TWAIN
2.0 introduces four new functions that are obtained from the Source Manager
through DAT_ENTRYPOINT.
TW_HANDLE PASCAL DSM_MemAllocate (TW_UINT32)
PASCAL DSM_MemFree (TW_HANDLE)
TW_MEMREF PASCAL DSM_MemLock(TW_HANDLE)
void PASCAL DSM_MemUnlock(TW_HANDLE)
These functions correspond to the
WIN32 Global Memory functions
mentioned in previous versions of the
TWAIN Specification: GlobalAlloc,
GlobalFree, GlobalLock, GlobalUnlock
On MacOS/X these functions call
NewPtrClear and DisposePtr. The lock
and unlock functions are no-ops, but
they still must be called. TWAIN 2.0
compliant Applications and Sources
must use these calls on all platforms
(Windows, MacOS/X and Linux). The
Source Manager takes the
responsibility to make sure that all
components are using the same memory
management API’s.
So to free resources you're supposed to call DSM_MemFree, which supposedly on Win32 platforms would be implemented through GlobalFree, or Marshal.FreeHGlobal.
Since this is mostly speculation on my part, you better validate with the specs of the specific TWAIN implementation you use.

It depends. Do you have any documentation (or source code) for the native functions you are calling?
Native code doesn't have a single deallocation function. This is one of the great advantages of the CLR.
If I was a betting man, I'd go for GlobalFree. But it's not going to be much fun trying various APIs until your code stops crashing.

Please show your unmanaged code. There are different ways to allocate memory in unmanaged land, and you must use the correct corresponding means of deallocating. You will probably end up implementing a Finalizer and IDisposable, and implementing the Dispose pattern as described here: http://www.codeproject.com/KB/cs/idisposable.aspx

Perhaps you could try creating a Bitmap object from hBitmap and then disposing it.
Bitmap bitmap = Bitmap.FromHBitmap(hBitmap);
bitmap.Dispose();

As everyone else is pointing out, It dpends on how it was allocated. However, if it really is a Win32 hbitmap, then you deallocate it with the Win32 "DeleteObject" function.

Marshal.AllocHGlobal VS Marshal.AllocCoTaskMem, Marshal.SizeOf VS sizeof()

I have the following struct:
[StructLayout(LayoutKind.Sequential, CharSet = CharSet.Auto)]
public struct WAVEHDR
{
internal IntPtr lpData; // pointer to locked data buffer
internal uint dwBufferLength; // length of data buffer
internal uint dwBytesRecorded; // used for input only
internal IntPtr dwUser; // for client's use
internal uint dwFlags; // assorted flags (see defines)
internal uint dwLoops; // loop control counter
internal IntPtr lpNext; // reserved for driver
internal IntPtr reserved; // reserved for driver
}
I need to allocate unmanaged memory to store an instance of above struct. A pointer to this struct will be passed to waveOut win32 api functions (waveOutPrepareHeader, waveOutWrite, waveOutUnprepareHeader).
Should I use Marshal.AllocHGlobal() or Marshal.AllocCoTaskMem()? What is the difference?
Should I pass sizeof(WAVEHDR) or Marshal.SizeOf(typeof(WAVEHDR)) to the memory allocation method? What is the difference?
NOTE that the allocated memory must be pinned.

A Windows program always has at least two heaps in which unmanaged memory is allocated. First is the default process heap, used by Windows when it needs to allocate memory on behalf of the program. The second is a heap used by the COM infrastructure to allocate. The .NET P/Invoke marshaller assumes this heap was used by any unmanaged code whose function signature requires de-allocating memory.
AllocHGlobal allocates from the process heap, AllocCoTaskMem allocates from the COM heap.
Whenever you write unmanaged interop code, you should always avoid a situation where code that allocates unmanaged memory is not the same as the code that frees it. There would be a good chance that the wrong de-allocator is used. This is especially true for any code that interops with a C/C++ program. Such programs have their own allocator that uses its own heap, created by the CRT at startup. De-allocating such memory in other code is impossible, you can't reliably get the heap handle. This is a very common source of P/Invoke trouble, especially because the HeapFree() function in XP and earlier silently ignore requests to free memory that wasn't allocated in the right heap (leaking the allocated memory) but Vista and Win7 crash the program with an exception.
No need to worry about this in your case, the mmsystem API functions you are using are clean. They were designed to ensure the same code that allocates also deallocates. This is one reason you have to call waveInPrepareHeader(), it allocates buffers with the same code that ultimately deallocates them. Probably with the default process heap.
You only need to allocate the WAVEHDR structure. And you are responsible for releasing it when you're done with it. The mmsystem APIs don't do it for you, most of all because they cannot do so reliably. Accordingly, you can use either allocator, you just need to make sure to call the corresponding free method. All Windows APIs work this way. I use CoTaskMemAlloc() but there really isn't a preference. Just that if I'm calling badly designed code, it is slightly likelier to use the COM heap.
You should never use sizeof() in an interop scenario. It returns the managed size of value type. That might not be the same after the P/Invoke marshaller has translated a structure type according to the [StructLayout] and [MarshalAs] directives. Only Marshal.SizeOf() gives you a guaranteed correct value.
UPDATE: there was a big change in VS2012. The C runtime library included with it now allocates from the default process heap instead of using its own heap. Long term, that makes AllocHGlobal the most likely avenue for success.

1) Marshal.AllocHGlobal will work for sure. Based on the docs for Marshal.AllocCoTaskMem, Marshal.AllocCoTaskMem should work too.
2) Use Marshal.SizeOf(typeof(WAVEHDR)). Although you can use the Marshal.SizeOf method, the value returned by this method is not always the same as the value returned by sizeof. Marshal.SizeOf returns the size after the type has been marshaled, whereas sizeof returns the size as it has been allocated by the common language runtime, including any padding.

2) As far as I know the sizeof can only be used with types that have a predefined size at the compile-time.
So use Marshal.SizeOf(typeof(WAVEHDR)).