We Keep Coding

Queue problems across multiple threads

There are many questions and articles on the subject of using a .NET Queue properly within a multi threaded application, however I can't find subject on our specific problem.
We have a Windows Service that receives messages onto a queue via one thread and is then dequeued and processed within another.
We're using lock when queuing and dequeuing, and the service had run fine for around 2 years without any problems. One day we noticed that thousands of messages had been logged (and so had been queued) but were never dequeued/processed, they seem to have been skipped somehow, which shouldn't be possible for a queue.
We can't replicate the circumstances that caused it as we have no real idea what caused it considering that day was no different from any of the others as far as we're aware.
The only idea we have is to do with the concurrency of the queue. We're not using the ConcurrentQueue data-type, which we plan on using in the hope it is a remedy.
One idea, looking at the source of the Queue type, is that it uses arrays internally, which have to be resized once these buffers have reached a certain length. We hypothesised that when this is being done some of the messages were lost.
Another idea from our development manager is that using multiple threads on a multicore processor setup means that even though locks are used, the individual cores are working on the data in their local registers, which can cause them to be working on different data. He said they don't work on the same memory and seems to think lock only works as expected one a single core processor using multiple threads.
Reading more about ConcurrentQueue's use of volatile I'm not sure that this would help, as I've read that using lock provides a stronger guarantee of threads using the most up-to-date state of memory.
I don't have much knowledge on this specific subject, so my question is whether the manager's idea sounds plausible, and whether we might have missed something that's required for the queue to be used properly.
Code snippet for reference (forgive the messy code, it does need refactoring):
public sealed class Message
{
public void QueueMessage(long messageId, Message msg)
{
lock (_queueLock)
{
_queue.Enqueue(new QueuedMessage() { Id = messageId, Message = msg });
}
}
public static void QueueMessage(string queueProcessorName, long messageId, Message msg)
{
lock (_messageProcessors[queueProcessorName]._queueLock)
{
_messageProcessors[queueProcessorName].QueueMessage(messageId, msg);
_messageProcessors[queueProcessorName].WakeUp(); // Ensure the thread is awake
}
}
public void WakeUp()
{
lock(_monitor)
{
Monitor.Pulse(_monitor);
}
}
public void Process()
{
while (!_stop)
{
QueuedMessage currentMessage = null;
try
{
lock (_queueLock)
{
currentMessage = _queue.Dequeue();
}
}
catch(InvalidOperationException i)
{
// Nothing in the queue
}
while(currentMessage != null)
{
IContext context = new Context();
DAL.Message msg = null;
try
{
msg = context.Messages.SingleOrDefault(x => x.Id == currentMessage.Id);
}
catch (Exception e)
{
// TODO: Handle these exceptions better. Possible infinite loop.
continue; // Keep retrying until it works
}
if (msg == null) {
// TODO: Log missing message
continue;
}
try
{
msg.Status = DAL.Message.ProcessingState.Processing;
context.Commit();
}
catch (Exception e)
{
// TODO: Handle these exceptions better. Possible infinite loop.
continue; // Keep retrying until it works
}
bool result = false;
try {
Transformation.TransformManager mgr = Transformation.TransformManager.Instance();
Transformation.ITransform transform = mgr.GetTransform(currentMessage.Message.Type.Name, currentMessage.Message.Get("EVN:EventReasonCode"));
if (transform != null){
msg.BeginProcessing = DateTime.Now;
result = transform.Transform(currentMessage.Message);
msg.EndProcessing = DateTime.Now;
msg.Status = DAL.Message.ProcessingState.Complete;
}
else {
msg.Status = DAL.Message.ProcessingState.Failed;
}
context.Commit();
}
catch (Exception e)
{
try
{
context = new Context();
// TODO: Handle these exceptions better
Error err = context.Errors.Add(context.Errors.Create());
err.MessageId = currentMessage.Id;
if (currentMessage.Message != null)
{
err.EventReasonCode = currentMessage.Message.Get("EVN:EventReasonCode");
err.MessageType = currentMessage.Message.Type.Name;
}
else {
err.EventReasonCode = "Unknown";
err.MessageType = "Unknown";
}
StringBuilder sb = new StringBuilder("Exception occured\n");
int level = 0;
while (e != null && level < 10)
{
sb.Append("Message: ");
sb.Append(e.Message);
sb.Append("\nStack Trace: ");
sb.Append(e.StackTrace);
sb.Append("\n");
e = e.InnerException;
level++;
}
err.Text = sb.ToString();
}
catch (Exception ne) {
StringBuilder sb = new StringBuilder("Exception occured\n");
int level = 0;
while (ne != null && level < 10)
{
sb.Append("Message: ");
sb.Append(ne.Message);
sb.Append("\nStack Trace: ");
sb.Append(ne.StackTrace);
sb.Append("\n");
ne = ne.InnerException;
level++;
}
EventLog.WriteEntry("Service", sb.ToString(), EventLogEntryType.Error);
}
}
try
{
context.Commit();
lock (_queueLock)
{
currentMessage = _queue.Dequeue();
}
}
catch (InvalidOperationException e)
{
currentMessage = null; // No more messages in the queue
}
catch (Exception ne)
{
StringBuilder sb = new StringBuilder("Exception occured\n");
int level = 0;
while (ne != null && level < 10)
{
sb.Append("Message: ");
sb.Append(ne.Message);
sb.Append("\nStack Trace: ");
sb.Append(ne.StackTrace);
sb.Append("\n");
ne = ne.InnerException;
level++;
}
EventLog.WriteEntry("Service", sb.ToString(), EventLogEntryType.Error);
}
}
lock (_monitor)
{
if (_stop) break;
Monitor.Wait(_monitor, TimeSpan.FromMinutes(_pollingInterval));
if (_stop) break;
}
}
}
private object _monitor = new object();
private int _pollingInterval = 10;
private volatile bool _stop = false;
private object _queueLock = new object();
private Queue<QueuedMessage> _queue = new Queue<QueuedMessage>();
private static IDictionary<string, Message> _messageProcessors = new Dictionary<string, Message>();
}

so my question is whether the manager's idea sounds plausible
Uhm. No. If all those synchronization measures would only work on single core machines, the world would have ended in complete Chaos decades ago.
and whether we might have missed something that's required for the queue to be used properly.
As far as your description goes, you should be fine. I would look at how you found out that you have that problem. logs coming in but then vanishing without being properly dequeued, wouldn't that be the default case if I simply turned off the service or rebooted the machine? Are you sure you lost them while your application was actually running?

You declare the object to be used for the lock as private object.
If you try this:
class Program
{
static void Main(string[] args)
{
Test test1 = new Test();
Task Scan1 = Task.Run(() => test1.Run("1"));
Test test2 = new Test();
Task Scan2 = Task.Run(() => test2.Run("2"));
while(!Scan1.IsCompleted || !Scan2.IsCompleted)
{
Thread.Sleep(1000);
}
}
}
public class Test
{
private object _queueLock = new object();
public async Task Run(string val)
{
lock (_queueLock)
{
Console.WriteLine($"{val} locked");
Thread.Sleep(10000);
Console.WriteLine($"{val} unlocked");
}
}
}
You will notice that the code that lies under the lock is executed even if another thread is running inside.
But if you change
private object _queueLock = new object();
To
private static object _queueLock = new object();
It changes how your lock works.
Now, this being your issue depends on if you have multiple instances that class or everything is running withing that same class.

GraceFul ShutDown in Azure Webjob in C#

i'm new to Azure Webjobs. I was trying to achieve the GraceFul Shutdown. while using the WebJobsShutdownWatcher Class.
Public static void Main()
{
try
{
var config = new JobHostConfiguration();
if (config.IsDevelopment)
{
config.UseDevelopmentSettings();
}
var watcher = new WebJobsShutdownWatcher();
Task.Run(() =>
{
bool isCancelled = false;
while (!isCancelled)
{
if (watcher.Token.IsCancellationRequested)
{
Console.WriteLine("WebJob cancellation Token Requested!");
isCancelled = true;
}
}
}, watcher.Token).Wait();
var host = new JobHost();
The following code ensures that the WebJob will be running continuously
host.RunAndBlock();
}
catch (Exception)
{
Console.WriteLine("Error");
}
}
To Achieve the GraceFul ShutDown, i have stop the Webjob and again Hosted on azure.
After Hosting on Azure, the Queues are not getting Trigger. when i Debug the Code the Control is Stop at the WebJobsShutdownWatcher Class.
What did i Done Wrong ?

As Thomas says, there are something wrong with your codes.
Since the WebJobsShutdownWatcher class will continue watch the webjobs status and you use wait method to wait the WebJobsShutdownWatcher class to get the cancel token, the host.RunAndBlock method is never hit.
You could remove the wait method, the codes will work well.
Here I write a test demo, it works well.
static void Main()
{
var config = new JobHostConfiguration();
if (config.IsDevelopment)
{
config.UseDevelopmentSettings();
}
var watcher = new WebJobsShutdownWatcher();
Task.Run(() =>
{
bool isCancelled = false;
while (!isCancelled)
{
if (watcher.Token.IsCancellationRequested)
{
Console.WriteLine("WebJob cancellation Token Requested!");
isCancelled = true;
}
}
}, watcher.Token);
var host = new JobHost(config);
// The following code ensures that the WebJob will be running continuously
host.RunAndBlock();
}

Single instance dotnetcore cli app on linux

I am interested in how to inforce a single instance policy for dotnetcore console apps. To my surprise it seems like there isn't much out there on the topic. I found this one stacko, How to restrict a program to a single instance, but it doesnt seem to work for me on dotnetcore with ubuntu. Anyone here do this before?

Variation of #MusuNaji's solution at: How to restrict a program to a single instance
private static bool AlreadyRunning()
{
Process[] processes = Process.GetProcesses();
Process currentProc = Process.GetCurrentProcess();
logger.LogDebug("Current proccess: {0}", currentProc.ProcessName);
foreach (Process process in processes)
{
if (currentProc.ProcessName == process.ProcessName && currentProc.Id != process.Id)
{
logger.LogInformation("Another instance of this process is already running: {pid}", process.Id);
return true;
}
}
return false;
}

This is a little more difficult on .NET core than it should be, due to the problem of mutex checking on Linux/MacOS (as reported above). Also Theyouthis's solution isn't helpful as all .NET core apps are run via the CLI which has a process name of 'dotnet' which if you are running multiple .NET core apps on the same machine the duplicate instance check will trigger incorrectly.
A simple way to do this that is also multi-platform robust is to open a file for write when the application starts, and close it at the end. If the file fails to open it is due to another instance running concurrently and you can handle that in the try/catch. Using FileStream to open the file will also create it if it doesn't first exist.
try
{
lockFile = File.OpenWrite("SingleInstance.lck");
}
catch (Exception)
{
Console.WriteLine("ERROR - Server is already running. End that instance before re-running. Exiting in 5 seconds...");
System.Threading.Thread.Sleep(5000);
return;
}

Here is my implementation using Named pipes. It supports passing arguments from the second instance.
Note: I did not test on Linux or Mac but it should work in theory.
Usage
public static int Main(string[] args)
{
instanceManager = new SingleInstanceManager("8A3B7DE2-6AB4-4983-BBC0-DF985AB56703");
if (!instanceManager.Start())
{
return 0; // exit, if same app is running
}
instanceManager.SecondInstanceLaunched += InstanceManager_SecondInstanceLaunched;
// Initialize app. Below is an example in WPF.
app = new App();
app.InitializeComponent();
return app.Run();
}
private static void InstanceManager_SecondInstanceLaunched(object sender, SecondInstanceLaunchedEventArgs e)
{
app.Dispatcher.Invoke(() => new MainWindow().Show());
}
Your Copy-and-paste code
public class SingleInstanceManager
{
private readonly string applicationId;
public SingleInstanceManager(string applicationId)
{
this.applicationId = applicationId;
}
/// <summary>
/// Detect if this is the first instance. If it is, start a named pipe server to listen for subsequent instances. Otherwise, send <see cref="Environment.GetCommandLineArgs()"/> to the first instance.
/// </summary>
/// <returns>True if this is tthe first instance. Otherwise, false.</returns>
public bool Start()
{
using var client = new NamedPipeClientStream(applicationId);
try
{
client.Connect(0);
}
catch (TimeoutException)
{
Task.Run(() => StartListeningServer());
return true;
}
var args = Environment.GetCommandLineArgs();
using (var writer = new BinaryWriter(client, Encoding.UTF8))
{
writer.Write(args.Length);
for (int i = 0; i < args.Length; i++)
{
writer.Write(args[i]);
}
}
return false;
}
private void StartListeningServer()
{
var server = new NamedPipeServerStream(applicationId);
server.WaitForConnection();
using (var reader = new BinaryReader(server, Encoding.UTF8))
{
var argc = reader.ReadInt32();
var args = new string[argc];
for (int i = 0; i < argc; i++)
{
args[i] = reader.ReadString();
}
SecondInstanceLaunched?.Invoke(this, new SecondInstanceLaunchedEventArgs { Arguments = args });
}
StartListeningServer();
}
public event EventHandler<SecondInstanceLaunchedEventArgs> SecondInstanceLaunched;
}
public class SecondInstanceLaunchedEventArgs
{
public string[] Arguments { get; set; }
}
Unit test
[TestClass]
public class SingleInstanceManagerTests
{
[TestMethod]
public void SingleInstanceManagerTest()
{
var id = Guid.NewGuid().ToString();
var manager = new SingleInstanceManager(id);
string[] receivedArguments = null;
var correctArgCount = Environment.GetCommandLineArgs().Length;
manager.SecondInstanceLaunched += (sender, e) => receivedArguments = e.Arguments;
var instance1 = manager.Start();
Thread.Sleep(200);
var manager2 = new SingleInstanceManager(id);
Assert.IsFalse(manager2.Start());
Thread.Sleep(200);
Assert.IsTrue(instance1);
Assert.IsNotNull(receivedArguments);
Assert.AreEqual(correctArgCount, receivedArguments.Length);
var receivedArguments2 = receivedArguments;
var manager3 = new SingleInstanceManager(id);
Thread.Sleep(200);
Assert.IsFalse(manager3.Start());
Assert.AreNotSame(receivedArguments, receivedArguments2);
Assert.AreEqual(correctArgCount, receivedArguments.Length);
}
}

The downside of deandob's solution is that one can launch the application from another path. So you may prefer some static path or a tmp path for all users.
Here is my attempt:
//second instance launch guard
var tempPath = Environment.GetEnvironmentVariable("TEMP", EnvironmentVariableTarget.Machine)
??
Path.GetTempPath();
var lockPath = Path.Combine(tempPath, "SingleInstance.lock");
await using var lockFile = File.OpenWrite(lockPath);
here I'm trying to get TEMP system variable at the scope of machine (not the user TEMP) and if its empty - fallback to the user's temp folder on windows or shared /tmp on some linuxes.

Continue the application even if a thirdparty application fails

I have a thirdparty exe file name Projtest.exe and using a c# console application written by myself for running the Projtest.exe through commandline.
i was able to succeed in writing the application,but my issue is that due to some unknown reasons Projtest.exe stops working and throws an error window.My application is stucking at this point.What i want is to continue my application if Projtest.exe throws an error window.How can i do that.Part of my code is given below.
try
{
var pro = new Process
{
StartInfo = {
Arguments = string.Format("Projtest.exe {0} {1} ", arg1, arg2)
}
};
pro.Start();
pro.WaitForExit();
var exit = pro.ExitCode;
}
catch (Exception ex)
{
Console.WriteLine(ex.ToString());
}

What you can to do is regularly check if 'pro' is responding say every 10 seconds.
You can do this simply with a timer or you could create a thread to do it.
Also it is worth checking for exit here as well so you don't block your application.
Here is an example with a timer.
static Timer appCheck = new Timer();
static Process pro;
static void Main(string[] args)
{
appCheck.Interval = (10000); // set timer interval in ms
appCheck.AutoReset = true;
appCheck.Elapsed += new ElapsedEventHandler(appCheck_Tick);
try
{
pro = new Process
{
StartInfo =
{
Arguments = string.Format(#"Projtest.exe"),
FileName = string.Format(#"Projtest.exe")
}
};
pro.Start(); // starts your program
appCheck.Start(); // starts the timer to keep a watch on your program
while (true) // this just keeps your console window open. if you use waitForExit your application will be blocked and the timer won't fire.
{
}
//pro.WaitForExit();
}
catch (Exception ex)
{
Console.WriteLine(ex.ToString());
}
}
static void appCheck_Tick(object sender, EventArgs e)
{
if (!pro.HasExited)
{
if (pro.Responding == false)
{
appCheck.Stop(); // stop the timer so you don't keep getting messages
Console.WriteLine("Not responding");
}
}
else
{
appCheck.Stop(); // stop the timer so you don't keep getting messages
Console.WriteLine("exited");
}
}

How to check if a thread is running in C #?

I created a thread in C # 4.0 and would like to know how do I check if it is running?

You can use Thread.IsAlive to check to see if a Thread is running.
That being said, if you're using C# 4, it's rarely a good idea to make "threads" manually. You should consider using the TPL and the Task/Task<T> class, as this provides a much cleaner model to attach work to run after the task completes, pull data out of the operation, etc.

I use Mutex to verify this. Sometimes just verify is Thread is alive with Thread.IsAlive is not safe if you are running on Background.
Try this:
private void btnDoSomething()
{
try
{
string nameThread = "testThreadDoSomething";
var newThread = new Thread(delegate() { this.DoSomething(nameThread); });
newThread.IsBackground = true;
newThread.Name = nameThread;
newThread.Start();
//Prevent optimization from setting the field before calling Start
Thread.MemoryBarrier();
}
catch (Exception ex)
{
}
}
public void DoSomething(string threadName)
{
bool ownsMutex;
using (Mutex mutex = new Mutex(true, threadName, out ownsMutex))
{
if (ownsMutex)
{
Thread.Sleep(300000); // 300 seconds
if (Monitor.TryEnter(this, 300))
{
try
{
// Your Source
}
catch (Exception e)
{
string mensagem = "Error : " + e.ToString();
}
finally
{
Monitor.Exit(this);
}
}
//mutex.ReleaseMutex();
}
}
}