I have a S#arp Architecture app that implements a lightweight queue-processing thing whereby various threads pull entities from a list and set their status to mark the fact that processing has started on those items.
Despite wrapping the start-processing bit in explicit transactions and using a C# lock(), I still get them starting at the same time sometimes.
Do I regret not using MSMQ ... well, yeah, but now this concurrency behaviour has got me baffled. Evidently there's something that I don't understand about NHibernate transactions and flushing. Can you help me out?
Here's the relevant bits of code:
private static object m_lock = new object();
private bool AbleToStartProcessing(int thingId)
{
bool able = false;
try
{
lock (m_lock)
{
this.thingRepository.DbContext.BeginTransaction();
var thing = this.thingRepository.Get(thingId);
if (thing.Status == ThingStatusEnum.PreProcessing)
{
able = true;
thing.Status = ThingStatusEnum.Processing;
}
else
{
logger.DebugFormat("Not able to start processing {0} because status is {1}",
thingId, thing.Status.ToString());
}
this.thingRepository.DbContext.CommitTransaction();
}
}
catch (Exception ex)
{
this.thingRepository.DbContext.RollbackTransaction();
throw ex;
}
if (able)
logger.DebugFormat("Starting processing of {0}",
thingId);
return able;
}
I would have expected this to guarantee that only one thread could change the status of a 'thing' at one time, but I get this in my logs pretty regularly:
2011-05-18 18:41:23,557 thread41 DEBUG src:MyApp.Blah.ThingJob - Starting processing of 78090
2011-05-18 18:41:23,557 thread51 DEBUG src:MyApp.Blah.ThingJob - Starting processing of 78090
.. and then both threads try and operate on the same thing and create a mess.
What am I missing? Thanks.
edit: changed code to reflect how my logging works in the real version
Setup concurrency in your NHibernate mappings, this post should help you get started.
http://ayende.com/blog/3946/nhibernate-mapping-concurrency
i think you are just crossed up on the status you are using to set that you are processing and to check that you are already processing. first one in sets ThingStatusEnum.Processing, but the next guy is checking for something different - ThingStatusEnum.PreProcessing. because ThingStatusEnum.Processing != ThingStatusEnum.PreProcessing, your locking means two threads are not
Related
I've created an app in C# .Net WPF. This app uses an sqlite database, but this database is shared with other programs.
So rarely my app and the others are running simultaneously, and more rarely they attempt to write in db simultaneously.. And All apps are crashing at this point...
So, I want to patch my app to wait that db isn't locked anymore before doing something with it.
I've been thinking about making a dirty try/catch loop with a number of attempting, but it seems to me to be a too dirty way (and waste of ressources)
The other program has a visual indicator when it uses db, so I've thought that a solution can involve user action. When database is locked, a MessageBox open to notify the user to wait until other program has finished before click ok and continue.
Is it a way to test if the database is locked without try/catch?
Following the comments, I tried a try/catch method. I'm still not convinced that this is the cleanest method, because I don't like the idea of waiting for an exception. An expected exception cannot be an exception in my opinion.
Maybe someone will come up with a solution that doesn't use this subterfuge, but I haven't found a better match for my expectations.
Thank you for your comments.
using SQLite;
public bool IsDatabaseLocked(string dbPath)
{
bool locked = true;
using (SQLiteConnection connection = new SQLiteConnection(dbPath))
{
try
{
connection.Execute("BEGIN EXCLUSIVE");
connection.Execute("COMMIT");
locked = false;
}
catch (SQLiteException)
{
// database is locked error
}
}
return locked;
}
public void WaitForDbToBeUnlocked(string dbPath)
{
int i = 0;
while (IsDatabaseLocked(dbPath))
{
i++;
if (i > 10)
{
MessageBox.Show("Please release manually Database or wait");
i = 0;
}
}
}
My answer is inspired by the following questions:
C# - How to detect if SQLite DB is locked?
I have a c++ dll function that i want to run inside the C# thread.
Some times I need to cancel that thread, and here is the issue :
Thread.Abort() is evil from the multitude of articles I've read on
the topic
The only way to do that was to use a bool and check it's value periodically.
My problem that even i set this value to true it didn't change and still equal to false in c++ code. However when I show a MessageBox that value changed and it works fine.
Any ideas why that value changed only when the MessageBox showed and please tell me how to fix that issue.
C#
public void AbortMesh()
{
if (currMeshStruct.Value.MeshThread != null && currMeshStruct.Value.MeshThread.IsAlive)
{
//here is my c++ Object and cancel mesh used to set bool to true;
MeshCreator.CancelMesh();
}
}
C++
STDMETHODIMP MeshCreator::CancelMesh(void)
{
this->m_StopMesh = TRUE;
return S_OK;
}
when I test the boolean value
if (m_StopMesh)
return S_FALSE;
The value here is always false even i call AbortMesh()
if (m_StopMesh)
return S_FALSE;
MessageBox(NULL,aMessage,L"Test",NULL);
if (m_StopMesh) // here the value is changed to true
return S_FALSE;
The non-deterministic thread abortion (like with Thread.Abort) is a really bad practice. The problem is that it is the only practice that allows you to stop your job when job does not know that it could be stopped.
There is no library or framework in .NET I know of that allows to write threaded code that could allow you to run an arbitrary task and abort it at any time without dire consequences.
So, you was completely write when you decided to use manual abort using some synchronization technique.
Solutions:
1) The simplest one is using of a volatile Boolean variable as it was already suggested:
C#
public void AbortMesh()
{
if (currMeshStruct.Value.MeshThread != null && currMeshStruct.Value.MeshThread.IsAlive)
{
MeshCreator.CancelMesh();
}
}
C++/CLI
public ref class MeshCreator
{
private:
volatile System::Boolean m_StopMesh;
...
}
STDMETHODIMP MeshCreator::CancelMesh(void)
{
this->m_StopMesh = TRUE;
return S_OK;
}
void MeshCreator::ProcessMesh(void)
{
Int32 processedParts = 0;
while(processedParts != totalPartsToProcess)
{
ContinueProcessing(processedParts);
processedParts++;
if (this->m_StopMesh)
{
this->MakeCleanup();
MessageBox(NULL,aMessage,L"Test",NULL);
}
}
}
Such code should not require any synchronization if you do not make any assumptions on completion of thread after the CancelMesh call - it is not instantaneous and may take variable amount of time to happen.
I don't know why the use of the volatile didn't help you, but there are few moments you could check:
Are you sure that the MeshCreator.CancelMesh(); method call actually happen?
Are you sure that m_StopMesh is properly initialized before the actual processing begins?
Are you sure that you check the variable inside the ProcessMesh often enough to have decent response time from your worker and not expecting something instantaneous?
2)Also if you use .NET 4 or higher you could also try to use the CancellationToken-CancellationTokenSource model. It was initially designed to work with Tasks model but works well with standard threads. It won't really simplify your code but taking into an account the async nature of your processing code will possibly simplify future integration with TPL
CancellationTokenSource cancTokenSource = new CancellationTokenSource();
CancellationToken cancToken = cancTokenSource.Token;
Thread thread = new Thread(() =>
{
Int32 iteration = 0;
while (true)
{
Console.WriteLine("Iteration {0}", iteration);
iteration++;
Thread.Sleep(1000);
if (cancToken.IsCancellationRequested)
break;
}
});
thread.Start();
Console.WriteLine("Press any key to cancel...");
Console.ReadKey();
cancTokenSource.Cancel();
3) You may want to read about interlocked class,monitor locks, autoresetevents and other synchronization, but they are not actually needed in this application
EDIT:
Well, I don't know how it couldn't help(it is not the best idea, but should work for such a scenario), so I'll try later to mock your app and check the issue - possibly it has something to do with how MSVC and CSC handle volatile specifier.
For now try to use Interlocked reads and writes in your app:
public ref class MeshCreator
{
private:
System::Boolean m_StopMesh;
...
}
STDMETHODIMP MeshCreator::CancelMesh(void)
{
Interlocked::Exchange(%(this->m_StopMesh), true);
return S_OK;
}
void MeshCreator::ProcessMesh(void)
{
Int32 processedParts = 0;
while(processedParts != totalPartsToProcess)
{
ContinueProcessing(processedParts);
processedParts++;
if (Interlocked::Read(%(this->m_StopMesh))
{
this->MakeCleanup();
MessageBox(NULL,aMessage,L"Test",NULL);
}
}
}
P.S.: Can you post the code that actually processes the data and checks the variable(I don't mean your full meshes calculations method, just its main stages and elements)?
EDIT: AT LEAST IT'S CLEAR WHAT THE SYSTEM IS ABOUT
It is possible that your child processes are just not exterminated quick enough. Read this SO thread about process killing.
P.S.: And edit your question to more clearly describe your system and problem. It is difficult to get the right answer to a wrong or incomplete question.
Try putting volatile before the field m_StopMesh:
volatile BOOL m_StopMesh;
I launched the c++ process using a thread and it worked fine.
If you want to communicate across process boundaries, you will need to use some sort of cross-process communication.
http://msdn.microsoft.com/en-us/library/windows/desktop/aa365574(v=vs.85).aspx
I find Named Pipes convenient and easy to use.
UPDATE
Your comment clarifies that the C++ code is running in-process.
I would suggest a ManualResetEvent. For a great overview of thread synchronization (and threads in general) check out http://www.albahari.com/threading/
I have following class that returns number of current Request per Second of IIS. I call RefreshCounters every minute in order to keep Requests per Second value refreshed (because it is average and if I keep it too long old value will influence result too much)... and when I need to display current RequestsPerSecond I call that property.
public class Counters
{
private static PerformanceCounter pcReqsPerSec;
private const string counterKey = "Requests_Sec";
public static object RequestsPerSecond
{
get
{
lock (counterKey)
{
if (pcReqsPerSec != null)
return pcReqsPerSec.NextValue().ToString("N2"); // EXCEPTION
else
return "0";
}
}
}
internal static string RefreshCounters()
{
lock (counterKey)
{
try
{
if (pcReqsPerSec != null)
{
pcReqsPerSec.Dispose();
pcReqsPerSec = null;
}
pcReqsPerSec = new PerformanceCounter("W3SVC_W3WP", "Requests / Sec", "_Total", true);
pcReqsPerSec.NextValue();
PerformanceCounter.CloseSharedResources();
return null;
}
catch (Exception ex)
{
return ex.ToString();
}
}
}
}
The problem is that following Exception is sometimes thrown:
System.InvalidOperationException: Category does not exist.
at System.Diagnostics.PerformanceCounterLib.GetCategorySample(String machine,\ String category)
at System.Diagnostics.PerformanceCounter.NextSample()
at System.Diagnostics.PerformanceCounter.NextValue()
at BidBop.Admin.PerfCounter.Counters.get_RequestsPerSecond() in [[[pcReqsPerSec.NextValue().ToString("N2");]]]
Am I not closing previous instances of PerformanceCounter properly? What am I doing wrong so that I end up with that exception sometimes?
EDIT:
And just for the record, I am hosting this class in IIS website (that is, of course, hosted in App Pool which has administrative privileges) and invoking methods from ASMX service. Site that uses Counter values (displays them) calls RefreshCounters every 1 minute and RequestsPerSecond every 5 seconds; RequestPerSecond are cached between calls.
I am calling RefreshCounters every 1 minute because values tend to become "stale" - too influenced by older values (that were actual 1 minute ago, for example).
Antenka has led you in a good direction here. You should not be disposing and re-creating the performance counter on every update/request for value. There is a cost for instantiating the performance counters and the first read can be inaccurate as indicated in the quote below. Also your lock() { ... } statements are very broad (they cover a lot of statements) and will be slow. Its better to have your locks as small as possible. I'm giving Antenka a voteup for the quality reference and good advice!
However, I think I can provide a better answer for you. I have a fair bit of experience with monitoring server performance and understand exactly what you need. One problem your code doesn't take into account is that whatever code is displaying your performance counter (.aspx, .asmx, console app, winform app, etc) could be requesting this statistic at any rate; it could be requested once every 10 seconds, maybe 5 times per second, you don't know and shouldn't care. So you need to separate the PerformanceCounter collection code from that does the monitoring from the code that actually reports the current Requests / Second value. And for performance reasons, I'm also going to show you how to setup the performance counter on first request and then keep it going until nobody has made any requests for 5 seconds, then close/dispose the PerformanceCounter properly.
public class RequestsPerSecondCollector
{
#region General Declaration
//Static Stuff for the polling timer
private static System.Threading.Timer pollingTimer;
private static int stateCounter = 0;
private static int lockTimerCounter = 0;
//Instance Stuff for our performance counter
private static System.Diagnostics.PerformanceCounter pcReqsPerSec;
private readonly static object threadLock = new object();
private static decimal CurrentRequestsPerSecondValue;
private static int LastRequestTicks;
#endregion
#region Singleton Implementation
/// <summary>
/// Static members are 'eagerly initialized', that is,
/// immediately when class is loaded for the first time.
/// .NET guarantees thread safety for static initialization.
/// </summary>
private static readonly RequestsPerSecondCollector _instance = new RequestsPerSecondCollector();
#endregion
#region Constructor/Finalizer
/// <summary>
/// Private constructor for static singleton instance construction, you won't be able to instantiate this class outside of itself.
/// </summary>
private RequestsPerSecondCollector()
{
LastRequestTicks = System.Environment.TickCount;
// Start things up by making the first request.
GetRequestsPerSecond();
}
#endregion
#region Getter for current requests per second measure
public static decimal GetRequestsPerSecond()
{
if (pollingTimer == null)
{
Console.WriteLine("Starting Poll Timer");
// Let's check the performance counter every 1 second, and don't do the first time until after 1 second.
pollingTimer = new System.Threading.Timer(OnTimerCallback, null, 1000, 1000);
// The first read from a performance counter is notoriously inaccurate, so
OnTimerCallback(null);
}
LastRequestTicks = System.Environment.TickCount;
lock (threadLock)
{
return CurrentRequestsPerSecondValue;
}
}
#endregion
#region Polling Timer
static void OnTimerCallback(object state)
{
if (System.Threading.Interlocked.CompareExchange(ref lockTimerCounter, 1, 0) == 0)
{
if (pcReqsPerSec == null)
pcReqsPerSec = new System.Diagnostics.PerformanceCounter("W3SVC_W3WP", "Requests / Sec", "_Total", true);
if (pcReqsPerSec != null)
{
try
{
lock (threadLock)
{
CurrentRequestsPerSecondValue = Convert.ToDecimal(pcReqsPerSec.NextValue().ToString("N2"));
}
}
catch (Exception) {
// We had problem, just get rid of the performance counter and we'll rebuild it next revision
if (pcReqsPerSec != null)
{
pcReqsPerSec.Close();
pcReqsPerSec.Dispose();
pcReqsPerSec = null;
}
}
}
stateCounter++;
//Check every 5 seconds or so if anybody is still monitoring the server PerformanceCounter, if not shut down our PerformanceCounter
if (stateCounter % 5 == 0)
{
if (System.Environment.TickCount - LastRequestTicks > 5000)
{
Console.WriteLine("Stopping Poll Timer");
pollingTimer.Dispose();
pollingTimer = null;
if (pcReqsPerSec != null)
{
pcReqsPerSec.Close();
pcReqsPerSec.Dispose();
pcReqsPerSec = null;
}
}
}
System.Threading.Interlocked.Add(ref lockTimerCounter, -1);
}
}
#endregion
}
Ok now for some explanation.
First you'll notice this class is designed to be a static singleton.
You can't load multiple copies of it, it has a private constructor
and and eagerly initialized internal instance of itself. This makes
sure you don't accidentally create multiple copies of the same
PerformanceCounter.
Next you'll notice in the private constructor (this will only run
once when the class is first accessed) we create both the
PerformanceCounter and a timer which will be used to poll the
PerformanceCounter.
The Timer's callback method will create the PerformanceCounter if
needed and get its next value is available. Also every 5 iterations
we're going to see how long its been since your last request for the
PerformanceCounter's value. If it's been more than 5 seconds, we'll
shutdown the polling timer as its unneeded at the moment. We can
always start it up again later if we need it again.
Now we have a static method called GetRequestsPerSecond() for you to
call which will return the current value of the RequestsPerSecond
PerformanceCounter.
The benefits of this implementation are that you only create the performance counter once and then keep using until you are finished with it. Its easy to use because you simple call RequestsPerSecondCollector.GetRequestsPerSecond() from wherever you need it (.aspx, .asmx, console app, winforms app, etc). There will always be only one PerformanceCounter and it will always be polled at exactly 1 times per second regardless of how quickly you call RequestsPerSecondCollector.GetRequestsPerSecond(). It will also automatically close and dispose of the PerformanceCounter if you haven't requested its value in more than 5 seconds. Of course you can adjust both the timer interval and the timeout milliseconds to suit your needs. You could poll faster and timeout in say 60 seconds instead of 5. I chose 5 seconds as it proves that it works very quickly while debugging in visual studio. Once you test it and know it works, you might want a longer timeout.
Hopefully this helps you not only better use PerformanceCounters, but also feel safe to reuse this class which is separate from whatever you want to display the statistics in. Reusable code is always a plus!
EDIT: As a follow up question, what if you want to performance some cleanup or babysitting task every 60 seconds while this performance counter is running? Well we already have the timer running every 1 second and a variable tracking our loop iterations called stateCounter which is incremented on each timer callback. So you could add in some code like this:
// Every 60 seconds I want to close/dispose my PerformanceCounter
if (stateCounter % 60 == 0)
{
if (pcReqsPerSec != null)
{
pcReqsPerSec.Close();
pcReqsPerSec.Dispose();
pcReqsPerSec = null;
}
}
I should point out that this performance counter in the example should not "go stale". I believe 'Request / Sec" should be an average and not a moving average statistic. But this sample just illustrates a way you could do any type of cleanup or "babysitting" of your PerformanceCounter on a regular time interval. In this case we are closing and disposing the performance counter which will cause it to be recreated on next timer callback. You could modify this for your use case and according the specific PerformanceCounter you are using. Most people reading this question/answer should not need to do this. Check the documentation for your desired PerformanceCounter to see if it is a continuous count, an average, a moving average, etc... and adjust your implementation appropriately.
I don't know, if this passes you .. I've read article PerformanceCounter.NextValue Method
And there was a comment:
// If the category does not exist, create the category and exit.
// Performance counters should not be created and immediately used.
// There is a latency time to enable the counters, they should be created
// prior to executing the application that uses the counters.
// Execute this sample a second time to use the category.
So, I have a question, which can lead to answer: isn't call to a RequestsPerSecond method happends too early?
Also, I would suggest you to to try check if the Category doesn't exists and log the info somewhere, so we can analyze it and determine which conditions we have and how often that happends.
I just solved this type of error or exception with:
Using,
new PerformanceCounter("Processor Information", "% Processor Time", "_Total");
Instead of,
new PerformanceCounter("Processor", "% Processor Time", "_Total");
I had an issue retrieving requests per second on IIS using code similar to the following
var pc = new PerformanceCounter();
pc.CategoryName = #"W3SVC_W3WP";
pc.InstanceName = #"_Total";
pc.CounterName = #"Requests / Sec";
Console.WriteLine(pc.NextValue());
This would sometimes throw InvalidOperationException and I was able to reproduce the exception by restarting IIS. If I run with a non warmed up IIS, e.g. after a laptop reboot or IIS restart, then I get this exception. Hit the website first, make any http request beforehand, and wait a second or two and I don't get the exception. This smells like the performance counters are cached,and when Idle they get dumped, and take a while to re-cache? (or similar).
Update1: Initially when I manually browse to the website and warm it up, it solves the problem. I've tried programmatically warming up the server with new WebClient().DownloadString(); Thread.Sleep() up to 3000ms and this has not worked? So my results of manually warming up server, might somehow be a false positive. I'm leaving my answer here, because it might be the cause, (i.e. manual warming up), and maybe someone else can elaborate further?
Update2: Ah, ok, here are some unit tests that summarises some learning from further experimenting I did yesterday. (There's not a lot on google on this subject btw.)
As far as I can reason, the following statements might be true; (and I submit the unit tests underneath as evidence.) I may have misinterpreted the results, so please double check ;-D
Create a performance counter and calling getValue before the category exists, e.g. querying an IIS counter, while IIS is cold and no process running, will throw InvalidOperation exception "category does not exist". (I assume this is true for all counters, and not just IIS.)
From within a Visual Studio unit test, once your counter throws an exception, if you subsequently warm up the server after the first exception, and create a new PerformanceCounter and query again, it will still throw an exception! (this one was a surprise, I assume this is because of some singleton action. My apologies I have not had enough time to decompile the sources to investigate further before posting this reply.)
In 2 above, if you mark the unit test with [STAThread] then I was able to create a new PerformanceCounter after one has failed. (This might have something to do with Performance counter possibly being singletons? Needs further testing.)
No pause was required for me before creating counter and using it, despite some warnings in MSDN same code documentation, other than the time it takes to create a performance counter itself before calling NextValue().In my case, to warm up the counter and bring the "category" into existance, was for me to fire one shot across the bow of IIS, i.e. make a single GET request, and viola, no longer get "InvalidOperationException", and this seems to be a reliable fix for me, for now. At least when querying IIS performance counters.
CreatingPerformanceCounterBeforeWarmingUpServerThrowsException
[Test, Ignore("Run manually AFTER restarting IIS with 'iisreset' at cmd prompt.")]
public void CreatingPerformanceCounterBeforeWarmingUpServerThrowsException()
{
Console.WriteLine("Given a webserver that is cold");
Console.WriteLine("When I create a performance counter and read next value");
using (var pc1 = new PerformanceCounter())
{
pc1.CategoryName = #"W3SVC_W3WP";
pc1.InstanceName = #"_Total";
pc1.CounterName = #"Requests / Sec";
Action action1 = () => pc1.NextValue();
Console.WriteLine("Then InvalidOperationException will be thrown");
action1.ShouldThrow<InvalidOperationException>();
}
}
[Test, Ignore("Run manually AFTER restarting IIS with 'iisreset' at cmd prompt.")]
public void CreatingPerformanceCounterAfterWarmingUpServerDoesNotThrowException()
{
Console.WriteLine("Given a webserver that has been Warmed up");
using (var client = new WebClient())
{
client.DownloadString("http://localhost:8082/small1.json");
}
Console.WriteLine("When I create a performance counter and read next value");
using (var pc2 = new PerformanceCounter())
{
pc2.CategoryName = #"W3SVC_W3WP";
pc2.InstanceName = #"_Total";
pc2.CounterName = #"Requests / Sec";
float? result = null;
Action action2 = () => result = pc2.NextValue();
Console.WriteLine("Then InvalidOperationException will not be thrown");
action2.ShouldNotThrow();
Console.WriteLine("And the counter value will be returned");
result.HasValue.Should().BeTrue();
}
}
Just out of curiousity, what do you have set for properties in Visual Studio? In VS go to Project Properties, Build, Platform target and change it to AnyCPU. I have seen it before where Performance Counters aren't always retrieved when it is set to x86, and changing it to AnyCPU could fix it.
I have a static class which is accessed by multiple remoting and other internal to the application threads. Part of the functionality of this class is controlling read/write access to various files, so I've implemented a static ReaderWriterLock on the list of files. The project uses the .net framework 2.0 as part of the customer requirements.
However when I stress test the system using a number of different clients (generally I'm using 16) each performing a large amount of reads and writes then very intermittently and only after several hours or even days have passed with at least 500k+ transactions completed the system crashes. Ok so we got a bug..
But when I check the logs of all locking events I can see that the following has happened:
1: Thread A acquires a write lock directly, checking IsWriterLock shows it to be true.
2: Thread B tries to acquire a reader lock and succeeds even though Thread A still has the write lock
3: System now crashes, stack trace now shows a null reference exception to the readerwriterlock
This process has been run several hundred thousand times previously with no errors and I can check the logs and see that the read lock was blocked in all cases previously until the write had exited. I have also tried implementing the readerwriterlock as a singleton but the issue still occurs
Has anybody ever seen anything like this before ??
A slimed down version of the readerwriterlock implementation used is shown below:
private const int readwriterlocktimeoutms = 5000;
private static ReaderWriterLock readerWriterLock = new ReaderWriterLock();
// this method will be called by thread A
public static void MethodA()
{
// bool to indicate that we have the lock
bool IsTaken = false;
try
{
// get the lock
readerWriterLock.AcquireWriterLock(readwriterlocktimeoutms);
// log that we have the lock for debug
// Logger.LogInfo("MethodA: acquired write lock; writer lock held {0}; reader lock held {1}", readerWriterLock.IsWriterLockHeld.ToString(),readerWriterLock.IsReaderLockHeld.ToString(), );
// mark that we have taken the lock
IsTaken = true;
}
catch(Exception e)
{
throw new Exception(string.Format("Error getting lock {0} {1}", e.Message, Environment.StackTrace));
}
try
{
// do some work
}
finally
{
if (IsTaken)
{
readerWriterLock.ReleaseWriterLock();
}
}
}
// this method will be called by thread B
public static void MethodB()
{
// bool to indicate that we have the lock
bool IsTaken = false;
try
{
// get the lock
readerWriterLock.AcquireReaderLock(readwriterlocktimeoutms);
// log that we have the lock for debug
// Logger.LogInfo("MethodB: acquired read lock; writer lock held {0}; reader lock held {1}", readerWriterLock.IsWriterLockHeld.ToString(),readerWriterLock.IsReaderLockHeld.ToString(), );
// mark that we have taken the lock
IsTaken = true;
}
catch (Exception e)
{
throw new Exception(string.Format("Error getting lock {0} {1}", e.Message, Environment.StackTrace));
}
try
{
// do some work
}
finally
{
if (IsTaken)
{
readerWriterLock.ReleaseReaderLock();
}
}
}
enter code here
#All finally have a solution to this problem. #Yannick you were on the right track...
If MSDN says that it's impossible to have reader and writer lock held at same time.
Today I got confirmation from microsoft that in cases of very heavy load on multiprocessor systems (note: I could never reproduce this problem on an AMD system only on Intel) its possible for ReaderWriterLock class objects to become corrupted, the risk of this is increased if the numer of writers at any given stage grows as these can backup in the queue.
For the last two weeks I've been running using the .Net 3.5 ReaderWriterLockSlim class and have not encountered the issue, which corresponds to what Microsoft have confirmed that the readerwriterlockslim class does not have the same risk of corruption as the fat ReaderWriterLock class.
If MSDN says that it's impossible to have reader and writer lock held at same time.
Is it possible in your process to have 2 readerWriterLock objects at any time, for some other reason ?
Another thing strange, is that Debugging a thread using isWriterLockHeld, whereas the current thread is a reader one, don't allow you to know about writing within another thread.
How do you know that Thread A still holds a writer lock, and how do you know that it's not the debug-Logging system that delay or "mix" instructions given by threads ?
Other thought, is it possible that other resource shared leads to a deadlock ? That would results somehow to a crash ? (while, Null Exception is still strange unless where consider the deadlock cleaned and readerWriterLock reset.
Your problem is strange, true.
And other question, that won't solve your problem. What do you use isTaken, whereas in debugging your application you rely on isWriterLockHeld (or isReaderLockHeld) ?
why not use it in your finally blocks ?
I have two instances running of same Windows Service. They check the health of each other and report if any issue is found. I have a critical job that needs to be performed so I am running it with a fail-over approach, it runs in Master, and if Master is not responding it runs in slave. This job needs to communicate over a specific serial port, I am trying to use Mutex to check for race condition. I dont have access to production, so before deploying I want to make sure my approach is fine. So please suggest if my use of Mutex is fine for the given case.
if (iAmRunningInSlave)
{
HealthClient hc = new HealthClient();
if (!hc.CheckHealthOfMaster())
return this.runJobWrapper(withMutex, iAmRunningInSlave);
else
return true; //master is ok, we dont need to run the job in slave
}
return this.runJobWrapper(withMutex, iAmRunningInSlave);
And then in runJobWrapper
bool runJobWrapper(bool withMutex, bool iAmRunningInSlave)
{
if (!withMutex)
return this.runJob(iAmRunningInSlave); //the job might be interested to know
Mutex mutex = null;
string mutexName = this.jobCategory + "-" + this.jobTitle; //this will be unique for given job
try
{
mutex = Mutex.OpenExisting(mutexName);
return false; //mutex is with peer, return false which will re-trigger slave
}
catch
{
try
{ //mean time mutex might have created, so wrapping in try/catch
mutex = new Mutex(true /*initiallyOwned*/, mutexName);
return this.runJob(iAmRunningInSlave); //the job might be interested to know where I am running
}
finally
{
if (null!=mutex) mutex.ReleaseMutex();
}
return false;
}
}
I had a similar issue recently.
The design of the Mutex class is a bit weird/different from the normal classes in .NET.
Using OpenMutex to check for an existing Mutex is not really nice as you have to catch an exception.
A better approach is to use the
Mutex(bool initiallyOwned, string name, out bool createdNew)
constructor, and check the value returned by createdNew.
You don't look to check the return value from runJobWrapper anywhere - is this intentional? It is not obvious what the return value actually means anyway. Also you really shouldn't catch each any every exception that OpenExisiting could possibly throw - Out of memory? Stack overflow? etc. etc. Just catch the one you mean to handle correctly.
Also your code looks to be somewhat fragile - I wouldn't be surprised if you have race conditions.
I noticed that mutex.ReleaseMutex() was not releasing the mutex immediately..I had to call GC.Collect()