I need to organise some lock when several threads get access to common resource at next code architecture:
There is one dll what has methods (marked with some attribute) which are called by reflection by external program. All this methods are called simultaneously by several threads from that external program.
Inside this methods we have references to other methods what are located in another dll. Inside that another we have methods from other dll.
External run program - (reflection) > dll -> dll -> dll
Everything was ok till the moment when all threads started go use the same resource (access to email box) at the last dll in a chain.
One try to delete message but another has already deleted it etc. In fact I do not care about it and can put try/catch if message does not exists but I do not want to use this solution.
I'm thinking about add some lock to read and delete operation and make this delete async way (I do not care about result and do not need to wait till it will be finished - just cleaning old messages) and use some code like below
lock(someObject)
{
//Access to email and delete old messages
}
But someObject has to be declared at the top level of that architecture (External run program) and this parameter has to be passed thought all chain. A lot of work and I do not want to do it.
Also I can put this lock at top program what run dll methods but a lot of changes need to be done too because I'll loose a lot of time because I ned to lock small part only but not all.
Any other suggestions?
An architectural suggestion would be creating a queue of operation accessing the mail resource, to avoid locking. This will loose a feedback for the caller, but it can be done asynchronously if needed. May be just the deleting phase has to be done enqueued, this depend on the exact job you are trying to do. Of course such a queue must be threaed safe, and depending on the level of reliability you want, can be a persistent queue ( like MSMQ )
I do think that the try catch solution you suggested is fine as long as there is no dependency with regard to order between threads (one thread creating a message, another thread adding an attachment to it). In that case the Queue as mentioned by Felice Pollano might be a good solution.
In the end, whether you lock or not, you'll need to handle conflicts. If these conflicts are predictable you could use locking otherwise try catch might be just fine.
Related
We have an old project that we are supporting and there is an issue that occurs most probably due to multi-threading.
The original implementer 'fixed' it by doing a Thread.sleep before executing the problematic section.
The workaround works but as the section is inside a loop the thread.sleep adds multiple minutes to the time it takes for the section to finish.
In the last month we have been we have been experimenting with lower values for the sleep but we wish to find the root cause. During our investigations we were doing lock on private objects wherever we felt like that would help.
We looked for anything that might be spawning additional threads - found none.
No Thread.start and no ThreadPool usage.
What is confusing us is that during debugging we find our main thread in the middle of about 8 other threads that we don't know who spawned them.
These are background threads so first thought I had was the threadpool but as I mentioned no mention of it in the code.
It is .net 2.0 so no Asyncs.
This is just a part of the bigger application so it is a windows service but we run it as CMD to be able to debug it easily The main application itself is a windows forms desktop app.
It also uses COM+ components if that is any help.
I've tried [STA] instead of [MTA].
Also Locking as aforementioned.
MemoryBarriers as well.
We still get the issue.
The issue is basically corrupted datasets and nulls in objects where they shouldn't be.
It happens in about once every 25-100 iterations so reproduction is not straight forward but we have devised a test specifically for this issue to try to reproduce it.
All that is pointing us into the direction of thread issues.
Back to the original question -
Who could possibly by spawning those additional threads and how do we prevent these threads for being created?
Please note the threads marked with red - those are background threads and as far as we can see no mention of them in the code.
The suspected thread in the screenshot is actively modifying the cols in the dataset. Problem is - the methods calling the SetColValueOnRow function that the thread is executing are typical and don't use any kind of threading.
The CPU affinity for this application is set to 1 Core [part of the original work-around]
Thanks
Edit: The database is oracle 12c but the issues we face happen before writing to the database.
They usually happen in DataSets where a whole record or a few of its columns can be wiped once every few testing iterations
I think you need to investigate why Thread.sleep works. It does not sound like the code is by itself spawning additional threads, but you would have to go through the entire code base to find that out - including the COM+ components.
So the first thing I would do is to start up the program in debug and just press the F10 key to step into the program. Then open up the threads debug window and see if you see about the same number of threads as given in your question. If you do, then those are simply threads from the thread pool and your issue is probably unrelated to the multiple threads.
If you don't see the same number of threads, then try setting a breakpoint at various stages of the program and see if you can find where those threads are getting created. When you find where they are getting created, you can try adding some locking at that point. But, your issue still might not be caused by multiple threads corrupting memory. You should investigate until you are convinced that the issue is due to multiple threads or something else.
I suspect that the issue might be related to one or more of the COM+ components or maybe the code is calling some long running database stored procedure. In any case, I suspect the reason why Thread.sleep works is because it is giving the suspect component enough time to complete its operation before starting on the next operation.
If this theory is true, then it suggests that there is some interaction between operations and when Thread.Sleep is given a sufficiently large value to allow the operation to complete - there are no interaction issues. This also suggests that perhaps one of the COM+ components is doing some things asynchronously. The solution might be to use locks or critical sections inside the COM+ components code. Another idea is to redesign the section of code that is causing the problem to allow multiple operations simultaneously.
So, the problem you are experiencing may not be due to multiple threads in the C# code you are looking at - but might be due to a long-running operation that will sometimes fail if not given sufficient time to complete before starting the next operation. This may or may not be due to multiple threads in the C# code.
I have a "Module" abstract base class which users inherit from to create dll's for a particular program. This base class exposes an abstract method "ProcessData" which does some processing on data and returns the processed data. The main program iterates over all the included dll's and calls everyone's "ProcessData" function. Kind of like this:
foreach (Module module in allModules)
{
Data newData = module.ProcessData(oldData);
//Do stuff with newData
}
There's a general "contract" that dll creators should abide by, which is that no file IO, web access, etc. should be performed in "ProcessData". However, some of these user modules have bugs which occur every once in a while in which they enter an infinite loop (or so I assume).
If I don't know what the heck these users are doing in their functions, is there still a way to halt their function execution if it's taking too long? I'd like my program to be robust enough to halt a module's execution if it detects that it's taking far too long, since right now the process thread just gets stuck forever. I've heard that starting a thread and then attempting "Abort" doesn't always work due to exception handling (among other things), but if I'm not in charge of the function, there must be some way for me to force it to stop. Hopefully....
You have several options ordered by reliability and robustness in that order.
Recommended way is you could create separate process and execute the hostile code there. Doing so will give you a chance to kill the process anytime you want.
You could create separate AppDomain and run the code in that new AppDomain, and you can choose to unload the AppDomain anytime you want. This is not robust as first option. But there is one.
You might be tempted to just call Thread.Abort but don't do that. It isn't doing what you expect it to do.
Related reading: Careful with that axe, part one: Should I specify a timeout?
I currently have an application which is basically a wrapper for ~10 "LongRunning" Tasks. Each thread should keep running indefinitely, but sometimes they lock up or crash, and sometimes the wrapper app spontaneously exits (I haven't been able to track that down yet). Additionally, the wrapper application can currently only be running for one user, and that user has to be the one to restart the threads or relaunch the whole app.
I currently have a monitor utility to let me know when the threads stop doing work so that they can be manually restarted, but I'd like to automatically restart them instead. I'd also like the wrapper to be available to everyone to check the status of the threads, and for the threads to be running even when the wrapper isn't.
Based on these goals, I think I want to separate the threads into a Windows Service, and convert the wrapper into something which can just connect to the service to check its status and manipulate it.
How would I go about doing this? Is this a reasonable architecture? Should I turn each thread into a separate service, or should I have a single multi-threaded service?
Edit: All the tasks log to the same set of output files (via a TextWriter.Synchronized(StreamWriter)), and I would want to maintain that behavior.
They also all currently share the same database connection, which means I need to get them all to agree to close the connection at the same time when it's necessary. However, if they were split up they could each use their own database connection, and I wouldn't need to worry about synchronizing that. I actually suspect that this step is one of the current failure points, so splitting it up would be a Good Thing.
I would suggest you to stay inside one multithreading service if possible. Just make sure that threads are handled correctly when Service Stop is triggered. Put brake flags inside blocks of code that will take a lot of time to execute. This way you will make your service responsive on Stop event. Log any exceptions and make sure to wait for all threads to exit until service is finally stopped. This will prevent you to run same "task" in multiple threads.
Maintaining one service is in the end easier then multiple services.
Splitting to multiple services would be reasonable if you require some separate functionalities that can run or not beside each other.
I don't think moving the threads to a Windows Service removes any of the problems. The service will still crash randomly and the threads will still exit randomly.
I assume that your long-running tasks implement a kind of worker loop. Wrap the body of that loop in a try-catch and log all exceptions. Don't rethrow them so that the task does not ever exit. Examine the logs to find the bugs.
Recently I worked with an external dll library where I have no influence on it.
Under some special circumstances, a method of this third party dll is blocking and never returning.
I tried to work around this issue by executing this method in a new AppDomain. After a custom timeout, I wanted to Unload the AppDomain and kill all this crap ;)
Unfortunately, it does not work - as someone would expect.
After some time it throws CannotUnloadAppDomainException since the blocking method does not allow aborting the thread gracefully.
I depend on using this library and it does not seem that there will be an update soon.
So can I work around this issue, even if it's not best practice?
Any bad hack appreciated :)
An AppDomain cannot typically solve that problem, it's only good to throw away the state of your program. The real issue is that your thread is stuck. In cases like these, calling Thread.Abort() is unlikely to work, it will just get stuck as well. A thread can only be aborted if it is a "alertable wait state", blocking on a CLR synchronization object. Or executing managed code. In a state that the CLR knows how to safely clean up. Most 3rd party code falls over like this when executing unmanaged code, no way to ever clean that up in a safe way. A decisive hint that this is the case is AppDomain.Unload failing to get the job done, it can only unload the AppDomain when it can abort the threads that are executing code in the domain.
The only good alternative is to run that code in a separate process. Which you can kill with Process.Kill(). Windows do the cleanup. You'd use a .NET interop mechanism to talk to that code. Like named pipes, sockets, remoting or WCF. Plus the considerable hassle of having to write the code that can detect the timeout, kills the process, starts it back up and recovers internal state since you now restart with an uninitialized instance of that 3rd party code.
Do not forget about the real fix. Create a small repro project that reproduces the problem. When it hangs, create a minidump of the process. Send both to the 3rd party support group.
after reading this (scroll down the end to Blocking Issues) I think your only solution is to run the method in a different process - this might involve quite a bit of refactoring and/or a 'host' project (eg Console application) that loads the method in question and makes it easy to call (eg reading args from command line) when launching the new process using the Process class
You can always use background worker, no need to create a new appdomain. This will ensure that you have complete control over the execution of the thread.
However, there is no way to ensure that you can gracefully abort the thread. As the dll is unmanaged, chances are there that it may cause memory leaks. However, spawning a new thread will ensure that your application does not crash when the Dll does not respond.
I'm creating a win service that monitors ftp logs, when a file has been uploaded I want to start an external application, like a powershell script, to do stuff with the file. my Question is do i want to spin this off into another thread when i do it or should I just wait until it finishes before moving on.
This process is already going to be in a thread (the service is going to be monitoring multiple servers) and the idea of threads starting threads worries me. Is this something to be worried about or is this a case of too much tinfoil in my hat.
Well, code it in a modular fashion and don't worry about threads. If, down the road, you find that your application will benefit from a multi-threaded approach then address it then. If you have build your components orthogonally then the threading part will fit more naturally.
Addressing threading concerns at the very beginning of an application always feel like premature optimization to me. Build the components first and worry about how to thread them later.
[Edit] I am in no way advising you to not think about threading at all. Every component needs to be build with the potential for use by multiple threads - this is a defensive and intelligent practice in all applications. What I meant was don't worry so much about how the application will handle threads and how to set up the thread management of the application first.
I think the more important question is what do you get out of spawning another thread? If you don't need to have the code execute in parallel, then don't do it. If you do, there should be no problem. If you are concerned with the child thread creating its own thread, then delegate the thread creation to the ThreadPool.
The primary question: do you need to know the outcome of that process? If you can fire and forget, then do that - it's easier. If you need the outcome, then wait for it.
Also, have you considered using the FileSystemWatcher? It works remotely.
Although somewhat off-topic, since you mentioned that you'll be launching a powershell script, I wanted to point out the option to run the script in-process via a powershell "runspace". Here's a minimal example:
using System.Management.Automation;
static class PoshExec
{
static void Exec(string scriptFilePath)
{
(new RunspaceInvoke()).Invoke("& " + scriptFilePath);
}
}
add a reference to
c:\Program Files\Reference Assemblies\Microsoft\WindowsPowerShell\v1.0\System.Management.Automation.dll