We Keep Coding

C# Pinvoke IntPtr in structs and memory

If i have understood right, when using structs with non blitable values, the struct data from unmanaged memory is copied into the managed memory (basically having the same struct twice).
Also if im not wrong, IntPtr variables, are stored in managed memory, but the data they point to is in the unmanaged memory.
Now lets say i have a delegate method, which is being called by a c++ function and receives a struct as ref, and a handler would be something like:
private void _handler(ref MyStruct p){}
Now the api says I should not keep the reference to the struct as the memory might be recycled and used for further calls, but since the unmanaged struct is copied into managed memory, and after I assign it to a local variable it is copied again (because it is a struct!) i shouldnt have any issues at all if the unmanaged memory gets freed or re written.
However, if the struct contains an IntPtr variable, i might save a copy of the pointer but not of the data it points, so if i try to access the IntPtr data from my saved struct, i might get some memory fault, is this correct?
One last question is, doing changes to the struct in managed memory, will also affect the unmanaged memory struct, since it is passed by ref, and it implicitly means IN/OUT, so after the event handler call ends, changes in managed memory will be made in the unmanaged memory as well?

Your understanding is largely correct. Taking away the details of blittabiltiy, the root question is one of ownership of the memory being passed to your callback. Typically, the contract between the caller of the callback and the callback itself is that the memory is owned by the caller and will be valid only for the duration of the callback itself. (Other contracts are possible as well - for instance, the caller could transfer ownership of the memory to the callback, in which case the callback is responsible for freeing the memory when it is done).
In the documentation you reference, the API is telling you not to keep a reference to the memory, which means that you are very likely in the standard case where the caller owns the structure. What this means is that both the contents of the structure and everything pointed to by that structure should be valid during the call to your callback but cannot be relied upon after your callback completes.
Copying a structure to a local wouldn't be problematic because the native memory should not be freed until after your callback completes, so keeping as many copies as you need during the execution of the callback should be fine. What would be problematic is saving a copy of the structure to a location, such as a static field, which will be alive after your callback returns.
Since the structure is implicitly in/out, then modifications you make to it will be reflected on the native side of the call as well. This is true regardless of if the structure contained only blittable data. If the structure contained non-blittable data then the CLR would marshal your changes to the structure back out to the native copy when your code completes.
Note that just because you can modify the native structure does not mean that the API contract expects you to do so. Frequently API signatures contain pointers to structures in C++ in order to avoid copying the entire structure when making a call, not to enable modification of the structure.

Does DangerousAddRef keep the whole SafeHandle alive or just the handle?

I am in the situation where I have a c# object containing an HGLOBAL and a List<Delegate>. The HGLOBAL holds references to the delegates and the HGLOBAL is registered in unmanaged code as a callback (specifically, a vtbl-style interface).
I'm trying to not get bogged down in the details of the overall design (I'm already expecting some screaming from those that understand what I'm doing in the previous paragraph). My question is currently one of lifetime management using functions I wish were documented differently.
I have converted the class holding the HGLOBAL and the List<Delegate> into a subclass of SafeHandle (because of CriticalFinalizerObject and because HGLOBAL is kinda a handle), and I do have the class that is instantiating this using CER-style atomic transactions to inform C# when the unmanaged code stops expecting it to stick around.
Does calling DangerousAddRef effectively bump up a reference count for the entire SafeHandle object? Will the List<Delegate> be included in this reference count? Another way of asking this is whether DangerousAddRef is effectively constructing a GCHandle to the entire SafeHandle object, or whether it is doing so simply to the SafeHandle.handle member inside?
If it is not applying to the object as a whole then the List<Delegate> is likely to be collected early and cause access violations when unmanaged code tries to call into it.
If it is applying to the object as a whole then I have just created a block of reference-counted objects within a garbage-collected heap and I had better be darned sure that I haven't created any reference loops.
I suspect it's the latter, but there are very few Google searches on this kinda stuff, and when the only symptoms you're dealing with are unmanaged access violations from the finalizer thread on process shutdown when no actual managed code is being ran then it starts to get difficult to see whether you're going in the right direction.

Determine managed vs unmanaged resources

There are lots of questions about managed vs unmanaged resources. I understand the basic definition of the two. However, I have a hard time knowing when a resource or object is managed or unmanaged.
When I think of unmanaged resources I tend to think of native code that isn't directly part of .NET such as pinvoke or marshaling resources. I would normally think of resources meant to interface to something that will use HW such as a file handle or network connection also being unmanaged.
What about .NET objects that wrap native unmanaged resources such as a FileStream.
A FileStream must use unmanaged resources, but when I implement the IDisposable pattern, should I consider this a managed or unmanaged resources?
I've been assuming thus far that if the object implements IDisposable, then it is managed. How would I know that IntPtr should be handled as an unmanaged resoruce?

A FileStream must use unmanaged resources, but when I implement the IDisposable pattern, should I consider this a managed or unmanaged resources?
A FileStream is a managed resource.
Managed resources are classes that contain (and must manage) unmanaged resources. Usually the actual resource is several layers down.
I've been assuming thus far that if the object implements IDisposable, then it is managed.
Correct.
How would I know that IntPtr should be handled as an unmanaged resoruce?
From the documentation of the API that you got its value from. But do note that in practice, most programmers never deal with unmanaged resources directly. And when you do have to, use the SafeHandle class to turn an unmanaged resource into a managed resource.

It is pretty straight-forward, you can never accidentally allocate an unmanaged resource. A pinvoke call is required to allocate it, you'd know about it. The term "object" is overloaded, but there is no such thing as an unmanaged object, all objects in a .NET program are managed. You may interop with code written in another language that supports creating objects, like C++. But you cannot directly use such an object, a C++/CLI wrapper is required. Which makes it a managed class that implements IDisposable.
If you work with a poorly documented library then do pay attention when you get an IntPtr back. That's a pretty strong indication that an unmanaged allocation is involved, either a pointer to unmanaged memory or an operating system handle. That library should then also give you a way to release it, if it doesn't otherwise manage it automatically. Contact the owner of the library if you are not sure how to properly deal with it.
It was Microsoft's job to provide managed wrapper classes around all common operating system resources. Like FileStream, Socket, etcetera. Those classes almost always implement IDisposable. The only thing you have to do in your code when you store such an class object in your own class is to implement IDisposable yourself, just so you call the Dispose() method on those object. Or use the using statement if you use them as a local variable in a method.

It is most helpful to think of a "resource" in this context as meaning "something which an object has asked something else to do on its behalf, until further notice, to the detriment of everyone else". An object constitutes a "managed resource" if abandoning it would result in the garbage collector notifying the object of abandonment, and the object in turn instructing anything that was acting on its behalf to stop doing so. An "unmanaged resource" is a resource which is not encapsulated within a managed resource.
If some object Foo allocates a handle to unmanaged memory, it asks the memory manager to grant it exclusive use of some area of memory, making it unavailable to any other code that might otherwise want to use it, until such time as Foo informs the memory manager that the memory is no longer needed and should thus be made available for other purposes. What makes the handle an unmanaged resource is not the fact that it was received via an API, but rather the fact that even if all deliberate references to it were abandoned the memory manager would forever continue granting exclusive use of the memory to an object which no longer needs it (and likely no longer exists).
While API handles are the most common kind of unmanaged resource, there are countless other kinds as well. Things like monitor locks and events exist entirely within the managed-code world of .net, but can nonetheless represent unmanaged resources since acquiring a lock and abandoning while code is waiting on it may result in that code waiting forever, and since a short-lived object which subscribes to an event from a long-lived object and fails to unsubscribe before it is abandoned may cause that long-lived object to continue carrying around the event reference indefinitely (a small burden if only one subscriber is abandoned, but an unbounded burden if an unbounded number of subscribers are created and abandoned).
Addendum
A fundamental assumption of the garbage collector is that when object X holds a reference to object Y, it is because X is "interested" in Y. In some situations, however, the reference may be held because X wants Y to hold a reference to it even though Y doesn't "care" one way or the other. Such situations occur frequently with notification event handlers. Object Y may want to be notified every time something happens to object X. Although X has to keep a reference to Y so it can perform such notifications, X itself doesn't care about the notifications. It only performs them because of a presumption that some rooted object might care about Y's receiving them.
In some cases, it's possible to use what's called a "weak event pattern". Unfortunately, while there are many weak event patterns in .net, all of them have quirks and limitations due to the lack of a proper WeakDelegate type. Further, while weak events are helpful, they're not a panacea. Suppose, for example, that Y has asked long-lived object X to notify it when something happens, the only existing reference to Y is the one X uses for such notification, the only thing Y does with such notification is to increment a property in some object Z, and that setting that property modifies nothing outside Z. Under that scenario, even though object Z will be the only thing in the universe that "cares" about object Y, Z won't hold any sort of reference to Y whatsoever, and so the garbage collector will have no way of tying Y's lifetime to that of Z. If a X holds a strong reference to Y, the latter will be kept alive even after nobody's interested in it. If X only holds a weak reference, then Y may be garbage-collected even if Z is interested in it. There is no mechanism by which the garbage collector can automatically infer that Z is interested in Y.

is it necessary to gchandle.alloc() each callback in a class?

I have a .NT class which has multiple delegates for callbacks from native code. Is it necessary to allocate all the delegates? I mean does GCHandle.Alloc() protects just the delegate or the entire class that owns the delegate from being collected?

A delegate has two relevant properties, Method and Target. The Target will be non-null if the delegate was created for an instance method. And that keeps the object alive, as long as the garbage collector can see the delegate instance.
Native code is relevant to having problems with callbacks. When you pass a delegate instance to a pinvoked native function then the P/Invoke marshaller will use Marshal.GetFunctionPointerForDelegate() to create a little stub that produces the required Target reference when the native code makes the callback. The garbage collector however can not see this stub and therefore won't find a reference to the delegate object. And collects it. The next callback from the native code produces a crash.
To avoid this, you must store the delegate object yourself so that it stays referenced for as long as the native code can make the callback. Storing it in a static variable is an obvious solution.

I am wrong.
You don't need to pinned the delegate (at the other hand, you cannot pinned the delegate, there will be a exception be thrown (System.ArgumentException: Object contains non-primitive or non-blittable data.)
reference:
http://social.msdn.microsoft.com/Forums/vstudio/en-US/bd662199-d150-4fbf-a5ee-7a06af0493bb/interop-pinning-and-delegates?forum=
Details about that in Chris Brumme's blog http://blogs.msdn.com/cbrumme/archive/2003/05/06/51385.aspx
Chris Brumme wrote:
Along the same lines, managed Delegates can be marshaled to unmanaged code, where they are exposed as unmanaged function pointers. Calls on those pointers will perform an unmanaged to managed transition; a change in calling convention; entry into the correct AppDomain; and any necessary argument marshaling. Clearly the unmanaged function pointer must refer to a fixed address. It would be a disaster if the GC were relocating that! This leads many applications to create a pinning handle for the delegate. This is completely unnecessary. The unmanaged function pointer actually refers to a native code stub that we dynamically generate to perform the transition & marshaling. This stub exists in fixed memory outside of the GC heap.
However, the application is responsible for somehow extending the lifetime of the delegate until no more calls will occur from unmanaged code. The lifetime of the native code stub is directly related to the lifetime of the delegate. Once the delegate is collected, subsequent calls via the unmanaged function pointer will crash or otherwise corrupt the process. In our recent release, we added a Customer Debug Probe which allows you to cleanly detect this – all too common – bug in your code. If you haven’t started using Customer Debug Probes during development, please take a look!

I think there will be something wrong happened (but not usually) when you just store the delegate object.
As we all know , the managed memory will be arranged by Garbage Collect. ( That means the physical memory address of a managed object will be changed. )
Imaging there is a long-time-life delegate to be called by native code, we set the delegate as the static member or class member . But sometime (we don't know when , we just know it will happen) , GC arranged memory, and the physical memory of the delegate may from 0x000000A to 0x0000010 . But the native code know nothing about it , for the native code , it only knows to call at 0x000000A forever.
So we should not only store the delegate object but also use GCHandle.Alloc to tell GC not move the physical memory of the delegate object. Then the native code will do well at callback time.
Well , because the GC do not arrange managed memory frequencly , so for a short-time-life delegate , even you do not call the GCHandle.Alloc ， your codes always "DO WELL" ， but sometimes it will mad.
Now , you know the reason.
reference:
http://dotnet.dzone.com/news/net-memory-control-use-gchandl

C# deallocate memory referenced by IntPtr

I am using some unmanaged code that is returning pointers (IntPtr) to large image objects. I use the references but after I am finished with the images, I need to free that memory referenced by the pointers. Currently, the only thing that frees the memory is to shutdown my entire app. I need to be able to free that memory from inside my application.
Here is the call to that allocates the memory. hbitmap is the pointer that is returned and needs to be deallocated.
[DllImport("twain_32.dll", EntryPoint = "#1")]
public static extern TwainResult DsImageTransfer(
[In, Out] Identity origin, [In] Identity dest, DataGroup dg,
DataArgumentType dat, Message msg, ref IntPtr hbitmap);

You need to use the specific memory allocator mechanism that was used to allocate the memory in the first place.
So, if you were using COM and the IMalloc interface to allocate the memory, then you have to pass the IntPtr back to the Free method on that implementation in order to free the memory allocated.
If you are indeed using the COM allocator that is returned by a call to CoGetMalloc, then you can call the static FreeCoTaskMem method on the Marshal class.
The Marshal class also has a method for freeing memory that is allocated through a call to LocalAlloc called FreeHGlobal.
However, and this is a common case, if the memory was allocated by the new operator in C++, or a call to malloc in C, then you have to expose a function in unmanaged code through interop which will free the memory appropriately.
In the case of C++, you would expose a function that takes a pointer and simply calls delete on that pointer. In the case of malloc, you would create a function that takes a pointer, and calls free on that pointer.
In specific regards to your question, it would seem that DsImageTransfer is a vendor-specific API (one that doesn't have much discoverability on the web either, I'm afraid), so more information is needed about that specific API function and how it allocates memory. Just knowing the handle type (an HBITMAP in this case) doesn't give any indication as to how it's allocated. It can be allocated with all the mechanisms mentioned above.
Assuming it is creating the HBITMAP using GDI Object api functions (specifically, the CreateBitmap function), then you could use the DeleteObject function to release the handle (as per the documentation page for the GDI Object API functions).

That would depend on how that memory was allocated. The Marshal class has methods for deallocating memory allocated through the common interop allocation patterns, like FreeCoTaskMem. If the unmanaged code uses a non Interop compatible way of allocating, then you cannot interop with it.
Updated
If I would venture a guess, the function #1 you invoke in twain_32.dll is the DS_ENTRY function in a TWAIN provider. The Twain specifications call out the memory resource management protocol:
Memory Management in TWAIN 2.0 and
Higher
TWAIN requires Applications and
Sources to manage each other’s memory.
The chief problem is guaranteeing
agreement on the API’s to use. TWAIN
2.0 introduces four new functions that are obtained from the Source Manager
through DAT_ENTRYPOINT.
TW_HANDLE PASCAL DSM_MemAllocate (TW_UINT32)
PASCAL DSM_MemFree (TW_HANDLE)
TW_MEMREF PASCAL DSM_MemLock(TW_HANDLE)
void PASCAL DSM_MemUnlock(TW_HANDLE)
These functions correspond to the
WIN32 Global Memory functions
mentioned in previous versions of the
TWAIN Specification: GlobalAlloc,
GlobalFree, GlobalLock, GlobalUnlock
On MacOS/X these functions call
NewPtrClear and DisposePtr. The lock
and unlock functions are no-ops, but
they still must be called. TWAIN 2.0
compliant Applications and Sources
must use these calls on all platforms
(Windows, MacOS/X and Linux). The
Source Manager takes the
responsibility to make sure that all
components are using the same memory
management API’s.
So to free resources you're supposed to call DSM_MemFree, which supposedly on Win32 platforms would be implemented through GlobalFree, or Marshal.FreeHGlobal.
Since this is mostly speculation on my part, you better validate with the specs of the specific TWAIN implementation you use.

It depends. Do you have any documentation (or source code) for the native functions you are calling?
Native code doesn't have a single deallocation function. This is one of the great advantages of the CLR.
If I was a betting man, I'd go for GlobalFree. But it's not going to be much fun trying various APIs until your code stops crashing.

Please show your unmanaged code. There are different ways to allocate memory in unmanaged land, and you must use the correct corresponding means of deallocating. You will probably end up implementing a Finalizer and IDisposable, and implementing the Dispose pattern as described here: http://www.codeproject.com/KB/cs/idisposable.aspx

Perhaps you could try creating a Bitmap object from hBitmap and then disposing it.
Bitmap bitmap = Bitmap.FromHBitmap(hBitmap);
bitmap.Dispose();

As everyone else is pointing out, It dpends on how it was allocated. However, if it really is a Win32 hbitmap, then you deallocate it with the Win32 "DeleteObject" function.