We Keep Coding

Keeping a COM object alive to queue longrunning tasks

From what I understand, COM objects can only be used in the thread they were instantiated in. If that thread dies, the object becomes invalid and you can't use it anymore. The error I face when I attempt to is COM object that has been separated from its underlying RCW cannot be used. The advice I've typically seen to deal with this is just to reinitialize a new COM object for every task you want to run.
The problem is that the COM object I'm using handles communication with another program, and re-establishing the connection with the same instance of that program with a new COM object is tricky. The operations I want to run are long winded and require feedback from the user in between, so including all the operations in a single Task is not really feasible afaik.
What I want to do is build a wrapper around that COM object that puts calls to it into a queue to be invoked by the appropriate thread, that would be compatible with async/await, and I would like to know if that's possible. But to avoid the XY problem I'll ask something different:
What is the most elegant way to keep a COM object functioning, off the main thread, so that I can continue to use it for multiple things?

COM object that has been separated from its underlying RCW cannot be used.
To keep the COM object alive you need to declare a corresponding RCW at a global scope, so the reference counter is not declared when the RCW is swiped from the heap. Also you need to pay attention to any Marshal.ReleaseComObject calls that may lead to the issue you faced with. This method is used to explicitly control the lifetime of a COM object used from managed code. You should use this method to free the underlying COM object that holds references to resources in a timely manner or when objects must be freed in a specific order.
For example, if you deal with Office applications like Outlook, it may detect cross-thread calls and throw exceptions in such cases.
The best solution is create a scheduler which can be called from secondary threads and queue such calls on the main thread to communicate with a COM server. Or just consider extracting all the required information (scalar value that don't involve COM objects) which can be simply consumed from secondary threads and then process it in the way you need.

An MTA Console application calling an STA COM object from multiple threads

Although there are many questions about COM and STA/MTA (e.g. here), most of them talk about applications which have a UI. I, however, have the following setup:
A console application, which is by default Multi-Threaded Apartment (Main() explicitly has the [MTAThread] attribute).
The main thread spawns some worker threads.
The main thread instantiates a single-threaded COM object.
The main thread calls Console.ReadLine() until the user hits 'q', after which the application terminates.
A few questions:
Numerous places mentions the need of a message pump for COM objects. Do I need to manually create a message-pump for the main thread, or will the CLR create it for me on a new STA thread, as this question suggests?
Just to make sure - assuming the CLR automagically creates the necessary plumbing, can I then use the COM object from any worker thread without the need of explicit synchronization?
Which of the following is better in terms of performance:
Let the CLR take care of the marshaling to and from the COM object.
Explicitly instantiate the object on a separate STA thread, and have other thread communicate with it via e.g. a ConcurrentQueue.

This is done automagically by COM. Since your COM object is single-threaded, COM requires a suitable home for the object to ensures it is used in a thread-safe way. Since your main thread is not friendly enough to provide such guarantees, COM automatically creates another thread and creates the object on that thread. This thread also automatically pumps, nothing you have to do to help. You can see it being created in the debugger. Enable unmanaged debugging and look in the Debug + Windows + Threads window. You'll see the thread getting added when you step over the new call.
Nice and easy, but it does have a few consequences. First off, the COM component needs to provide a proxy/stub implementation. Helper code that knows how to serialize the arguments of a method call so the real method call can be made on another thread. That's usually provided, but not always. You'll get a hard to diagnose E_NOINTERFACE exception if it is missing. Sometimes TYPE_E_LIBNOTREGISTERED, a common install problem.
And most significantly, every call on the COM component will be marshaled. That's slow, a marshaled call is usually around 10,000x slower than a direct call on a method that itself takes very little time. Like a property getter call. That can really bog your program down of course.
An STA thread avoids this and is therefore the recommended way to use a single-threaded component. And yes, it is a requirement for an STA thread to pump a message loop. Application.Run() in a .NET program. It is the message loop that marshals calls from one thread to another in COM. Do note that it doesn't necessarily mean that you must have a message loop. If no call ever needs to marshaled, or in other words, if you make all the calls on the component from the same thread, then the message loop isn't needed. That's typically easy to guarantee, particularly in a console mode app. Not if you create threads yourself of course.
One more nasty detail: a single-threaded COM component sometimes assumes it is created on a thread that pumps. And will use PostMessage() itself, typically when it uses worker threads internally and needs to raise events on the STA thread. That will of course not work correctly anymore when you don't pump. You normally diagnose this by noticing that events are not being raised. The common example of such a component is WebBrowser. Which heavily uses threads internally but raises events on the thread on which it was created. You'll never get the DocumentCompleted event if you don't pump.
So putting [STAThread] on your Main() method might be enough to get happy fast code, even without a call to Application.Run(). Just keep the consequences in mind, seeing a method call deadlock or an event not getting raised is the tell-tale sign that pumping is required.

Yes, it is possible to create a STA COM object from an MTA thread.
In this case, COM (not CLR) will create an implicit STA apartment (a separate COM-owned thread) or re-use the existing one, created ealier. The COM object will be instantiated there, then a thread-safe proxy object (COM marshalling wrapper) will be created for it and returned to the MTA thread. All calls to the object made on the MTA thread will be marshalled by COM to that implicit STA apartment.
This scenario is usually undesirable. It has a lot of shortcomings and may simply not work as expected, if COM is unable to marshal some interfaces of the object. Check this question for more details. Besides, the message pump loop, run by the implicit STA apartment, pumps only a limited number of COM-specific messages. That may also affect the functionality of the COM.
You may try it and it may work well for you. Or, you may run into some unpleasant issues like deadlocks, quite difficult to diagnose.
Here is a closely related question I just recently answered:
StaTaskScheduler and STA thread message pumping
I'd personally prefer to manually control the logic of the inter-thread calls and thread affinity, with something like ThreadAffinityTaskScheduler proposed in my answer.
You may also want to read this: INFO: Descriptions and Workings of OLE Threading Models, highly recommended.

Do I need to manually create a message-pump for the main thread,
No. It is in the MTA therefore no message pump is needed.
or will the CLR create it for me on a new STA thread
If COM creates the thread (because there is no STA in the process) then it also creates the message pump (and a hidden window: can be seen with the SPY++ and similar debugging tools).
COM object from any worker thread without the need of explicit synchronization
Depends.
If the reference to the single threaded object (STO) was created in the MTA then COM will supply the appropriate proxy. This proxy is good for all threads in the MTA.
In any other case the reference will need to be marshalled to ensure it has the correct proxy.
is better in terms of performance
The only answer to this is to test both and compare.
(Remember if you create the thread for the STA and then instantiate the object locally you need to do the message pumping. It is not clear to me that there is any CLR level lightweight message pump—including WinForms just for this certainly isn't.)
NB. The only in depth explanatory coverage of COM and the CLR is .NET and COM: The Complete Interoperability Guide by Adam Nathan (Sams, January 2002). But it is based on .NET 1.1 and now out of print (but there is a Kindle edition and is available via Safari Books Online). Even this book doesn't describe directly what you are trying to do. I would suggest some prototyping.

Releasing COM objects in a background thread

Extension to Release COM Object in C#
I noticed that saving the MailItem and releasing is a time consuming task. So, it is safe to do the following ? (pseudo-code below)
Thread 1 (main thread)
- Open 10 (different .msg files) - MailItems [List<MailItem> items]
- user works on them and want to save and close all of them with one click.
- On_save_All_click (runs on main thread)
- Do
- toBeClearedList.addAll(items);
- items.clear() [so that main thread cannot access those items]
- BG_Thread.ExecuteAsyn(toBeClearedList);
- End
Thread 2 (background thread) (input - List<MailItems>)
- foreach(MailItem item in input)
item.save();
System.Runtime.InteropServices.Marshal.ReleaseComObject(item)
- done
I wrote couple of tests and looks like its working; Just want to know if its safe to do so ? "Releasing COM objects in different thread than the one in which it was created"
Thanks
Karephul

When using COM from unmanaged code (C/C++), the rules are pretty strict: you can only call methods on an interface from the same apartment that you acquired the object on. So if you obtain an interface pointer on an STA thread, then only that thread is allowed to call any of the methods. If you obtain an interface pointer on an MTA thread, then only other threads in the same MTA can use that pointer. Any other use that crosses apartments requires that the interface pointer is marshaled to the other apartment.
However, that's unamanged world. .Net adds a whole layer on top of COM which buries a lot of these low-level details, and for the most part, once you get your hands on an interface, you can pass that interface around between threads as much as you please without having to worry about the old threading rules. What's happening here is that it are actually passing around references to an object called a "Runtime Callable Wrapper" (RCW), and it is managing the underlying COM interface, and controlling access to it accordingly. (It takes on the burden of upholding the COM apartment rules so that you don't have to, and that's why it can appear that the old COM threading rules don't apply in .Net: they do, they're just hidden from you.)
So you can safely call Release or other methods from another thread: but be aware that if the original thread was STA, then calling those methods will cause the underlying RCW to marshal the call back to the original owning thread so that it still upholds the underling COM rules. So using a separate thread may not actually get you any performance in the end!
Some articles worth reading that fill in some of the details here:
The mapping between interface pointers and runtime callable wrappers (RCWs) - good overview of some of the details of how RCW's work, but it doesn't say much about threading.
Improving Interop Performance - Marshal.ReleaseComObject - some good notes on when to use or not use ReleaseComObject, and how it works with the RCWs.
cbrumme's WebLog - Apartments and Pumping in the CLR - more internals than you're likely to want to know; this is a couple of CLR releases out of date, but still gives a good insight into how .Net covers over many of the underlying COM issues.

If I remember my COM properly (I always hated that technology, way too complicated), if the COM object is single-threaded (that is, belongs in a single-threaded-apartment), releasing it from another thread isn't going to do you any good - it's just going to execute the actual release code in your main thread.
You are going to see a little difference between your code, and releasing the objects from the main thread in one loop. If you release the objects in one loop, your UI is going to be unresponsive until you release all messages. By using a secondary thread, your UI thread will release one message, then handle other events, then release another, and so on. You can get the same effect by sending yourself a message (or using a Dispatcher if you have a WPF application), and avoid having another thread.

Disposing of COM in multi-threaded environment

I'm trying to properly dispose of a legacy VFP (FoxPro) COM control wrapped in a RCW generated by Visual Studio. The control exposes a Destroy method I should call to allow the control to properly tear itself down. There is a very good chance a method on the control may be executing on a background thread when a request is made to dispose of the COM instance. VFP is a single-threaded apartment model, so when calling Destroy it should just be added to the VFP execution stack.
Calling Destroy would ideally be the right thing to do as it allows the COM instance to clean up some resources. My concern is that instantiating a VFP COM control actually starts up a VFP language runtime instance that the control is hosted in and that instance may be locked up (non-responsive). This COM component exposes functionality in a large enterprise-scale 20-year-old legacy app and I have seen situations where a .NET thread attempting to call a method on this control simply blocks without throwing an error (always caused by bugs in the legacy VFP code). This doesn't happen often, but it is often enough that it prompted me to build an instance manager that runs methods on the VFP COM instance in a background thread and periodically checks to see if that thread is blocked, and if so, destroys the COM instance and thread and restarts a new instance to monitor.
Is this the right way to dispose of the thread that a background method may be executing on?
Should I attempt to get fancier by trying to call the Destroy method to allow the COM control to properly tear down?
if (_vfpThread != null)
{
try
{
if (_vfpThread.IsAlive)
_vfpThread.Abort();
}
catch (ThreadAbortException)
{ }
finally
{
_vfpThread = null;
}
}
if (_vfpInstance != null)
{
Marshal.ReleaseComObject(_vfpInstance);
_vfpInstance = null;
}

When a method call is pending on a VFP-based COM object (which always runs in an STA apartment), invoking any method on that same COM object from another thread will block until the former call returns (exits the apartment).
That means, any thread attempting to call Destroy() concurrently will be at the mercy of that first thread. And if that thread doesn't know to exit voluntarily, it could in theory keep the disposing thread blocked indefinitely. So, in other words, there's no direct way to ask the 1st thread to exit the method immediately by calling another method on the COM object from within another thread. Calling _vfpThread.Abort() should work, but the safety of this approach largely depends on the internals of the VFP class.
In many cases, due to it being legacy code, it won't have anything like a try/catch/finally section that would allow for a graceful exit, therefore resources may wind up being left unreleased. - BAD!
Another approach would be to set an external flag somewhere (registry, file, whatever), which would be available for reading by that 1st thread from within the method it is executing. That of course requires that the VFP class be aware of having to read the flag from each of its COM-published methods, and act accordingly and quickly.
Also, regarding your code snippet.
Catching ThreadAbortException in the code that is aborting a thread only makes sense if the thread executing this code is aborting itself. Which would be pretty awkward, since it could instead just return from the method. (Or, is this thread that is calling _vfpThread.Abort() also potentially being aborted from yet another thread?)
In a normal scenario, what you'd need wrapped in a ThreadAbortException catcher is the 1st thread's main code that performs calls to all those business methods on the COM object.
Ideally, you'd have this as deep down the stack as the VFP methods themselves where the code would be able to gracefully close all resources/tables etc before re-throwing the exception.
And then in the main method that you passed to the ThreadStart, you'd have a similar catcher except it would peacefully return from the method, thereby terminating the thread or releasing it to the thread pool.

Yes, I did understand your code correctly.-Thanks.
Aborting the vfp thread if it does not exit gracefully within 60 seconds is perhaps the only thing you could do.
In terms of what Dispose should do - it should try its best to release all unmanaged resources, which are unfortunately hidden from this code as they are used/opened from within the VFP COM class. So, if the COM object is seized, the main thread won't be able to force it to release those resources. Perhaps what you could try doing is wrapping the entire body of the COM business method in a VFP try-catch block and releasing resources/closing tables in the catch section. There's a good chance that the try-catch block would capture the ThreadAbortException caused by calling _vfpThread.Abort() from the main thread.

C# Managed Thread Cleanup

After my application creates a thread using a ParameterizedThreadStart delegate, that thread performs some initialization and runs to completion. Later on, I can observe that this thread is no longer active because its IsAlive property is false and ThreadState property is ThreadState.Stopped.
Once a thread reaches this state they remain in my application, still existing as thread objects until my application shuts down. Are there any steps I can take to dispose of them once they're no longer active? I would like to delete the object and deallocate any resources so that any given moment the only thread objects I have are active threads. Thread doesn't implement IDisposable, though, so I'm not sure how I should do this.

You're holding onto the reference to the thread in your code.
If you have written code that will check the state of the thread, then that code inherently will keep the thread object alive until the GC collects it.
Once you are finished with a thread, or ideally if you don't need to access it, make sure you null all references to it. Thread doesn't implement IDisposable because as you've made clear this wouldn't make sense for a thread.
Threads are native in .Net so you don't have to worry about leaks. If you're certain they will stop then just delete them from your list once you are sure it has finished.

It sounds like you need to let go of your reference to the Thread object, so the garbage collector can discard it. Just set the reference you have to null, and let the GC do its job when it's ready.
Depending on your situation, you may wish to use a WeakReference (or my friend Cyrus' WeakReference<T>).

Is the unmanaged thread still there, did the thread actually return from its ParameterizedThreadStart method? Also try making IsBackground = false