We Keep Coding

Alternative for BlobContainerClient.CreateIfNotExistsAsync method to avoid 409 http error

Prerequisites
WindowsAzure.Storage V 9.3.3
Azure.Storage.Blobs V 12.6.0
Datadog APM monitoring
Context:
This is a known problem, that a blob client CreateIfNotExists(Async) returns 409 during execution, in case if a container already has been created before
Due to existed implementation, the CreateIfNotExists check is executing per a request, that mean a lot of times
As a result, there are a lot of 409 errors in a log & monitor systems
Also, it's kind of challenge to move logic to the app root, and, thus, to avoid per request CreateIfNotExists check
The possible fix, under consideration:
To mitigate the issue, I gonna to replace it with something like:
if (!await blobClient.ExistsAsync().ConfigureAwait(false))
{
await containerClient.CreateAsync().ConfigureAwait(false);
}
Question: And one moment that bothering me:
Could be such replacement become a concern within high concurrent workflow ?
P.s I've looked into CreateIfNotExistsAsync implementation at 'Azure.Storage.Blobs' nuget. From my point of view, nothing special about concurrency, but maybe I'm wrong. Pls. share you experience

Your code will have to handle this problem. Checking for existence won't help.
The problem is concurrency on the server side, not concurrency in the client. The source shows thatCretaeIfNotExists already handles the case of an existing container by simply ignoring the exception. One could ask why an exception is thrown instead of checking the status code, but the code doesn't throw a 409 if a container exists:
try
{
response = await CreateInternal(
publicAccessType,
metadata,
encryptionScopeOptions,
async,
cancellationToken,
operationName)
.ConfigureAwait(false);
}
catch (RequestFailedException storageRequestFailedException)
when (storageRequestFailedException.ErrorCode == BlobErrorCode.ContainerAlreadyExists)
{
response = default;
}
catch (Exception ex)
{
ClientConfiguration.Pipeline.LogException(ex);
scope.Failed(ex);
throw;
}
finally
{
ClientConfiguration.Pipeline.LogMethodExit(nameof(BlobContainerClient));
scope.Dispose();
}
return response;
For a conflict to occur, two CreateInternal calls would have to try to create the same non-existent container. That's why checking for existence won't help here. Even if an existence check was used, both clients would still try to execute Create at the same time and both would get a 409.
The application will have to handle this case. The best option would be reducing the chance of conflicts by not making this call on every request. This could be done by using eg a Lazy<> or asynchronous Lazy to make this call. Or using queued workers, reducing the number of calls that can happen at the same time.
The application would still have to handle the occasional 409. One way would be to simply retry CreateIfNotExists after a delay, until the call succeeds. That's better than calling Exists because the container's creation could fail.

BlobContainerClient containerClient = blobServiceClient.GetBlobContainerClient(containerName);
if (!containerClient.Exists())
{
containerClient = await blobServiceClient.CreateBlobContainerAsync(containerName, PublicAccessType.BlobContainer);
}

Set StackyClient Timeout

I use the StackyClient library to loop over newest questions in Stack Overflow. Generally this works:
StackyClient client = new StackyClient("1.1", "", Sites.MetaStackOverflow, new UrlClient(), new JsonProtocol());
var o = new QuestionOptions();
...
IPagedList<Question> l = null;
while (!stop) {
l = client.GetQuestions(o);
var newQuestions = (from Question q in l
orderby q.Id descending
select q).ToList();
...
}
But at some point this seems to fail. Maybe due to network connectivity problems or I don't know what. The program hangs at client.GetQuestions(o).
I would like to detect that problem and reconnect again. I thought about setting a timeout somehow to prevent the program from hanging. How can it be done?

There are two possible answers here:
Are you sure this is hanging? Did you run it under a debugger and make sure it's not throwing an uncaught exception? Did you check with ProcessExplorer and/or Fiddler to see that the request and socket are still open? It seems more likely to me that the socket is being closed or you're getting an exception and it's not being caught.
If it is confirmed that this is hanging at GetQuestions, then a review of the code indicates that you'll have to ask the author of Stacky to implement this feature or you're going to have to implement it yourself... the feature does not exist today, from what I can tell.
In Stacky's UrlClient.cs there is a blocking call to HttpWebRequest.GetResponse and there is a blocking call to ReadToEnd on the response stream. No timeout is being set for either call, nor on the HttpWebRequest object itself:
Stacky/UrlClient.cs
In addition, I don't see anything in the connectionManagement settings that would allow setting a timeout on responses and reading from the response stream of HttpWebRequest:
connectionManagement setting
The Stack UrlClient code is re-throwing WebExceptions other than 404 and 407 (proxy auth required), so if you got a 408 (the server timing you out) from StackOverflow it would throw out of the Stacky GetQuestions call, which is why I suspect it might be that you're seeing an uncaught exception and not a hang.
I periodically see 408's from StackOverflow when just browsing questions in Chrome, so it is possible that this is happening to your program and you just need to catch the exception and retry after a delay.
Hope this helps - Harold

handling errors through the async dll stack

I'm trying to handle errors that are passed through 2 dlls I've created. So Console.exe calls dll 1. dll 1 completes an async MQ message read and the handler calls dll 2. If dll 2 errors it passes the Exception (throw) without a problem. But the dll 1 (async) handler catch the throw from dll 2 and give me an unhandled by user message.. I have followed the msdn code to add in the IAsyncResult to keep the hander alive but the issue persists.
can anyone advise on how I should handle this stack and get the handler error returned to the console.exe program so I can present it to the user. Code below:-
Console.exe (snippet)
try
{
_msmq.MSMQ_GetMessage(_msgPath);
//set up the print of the number of queue messages
Console.WriteLine("Main thread: starting a timer");
Timer t = new Timer(ComputeBoundOp, _msgPath, 0, 2000);
Console.Write("Press any key to continue . . .");
Console.ReadKey(true);
t.Dispose(); // Cancel the timer now
}
catch (MessageQueueException _msgQex)
{
Console.WriteLine("An error occurred with the queue:- " + _msgQex);
}
catch (Exception _ex)
{
Console.WriteLine("An error occurred with the queue:- " + _ex);
}
dll 1
public void MSMQ_GetMessage(string _MQ_Path)
{
try
{
//set the correct message queue
MessageQueue _msgQ = new MessageQueue(_MQ_Path, QueueAccessMode.ReceiveAndAdmin);
//set the format of the message queue
_msgQ.Formatter = new XmlMessageFormatter(new Type[] { typeof(_TwitterStreamFeed) });
_msgQ.ReceiveCompleted += new ReceiveCompletedEventHandler(_msgQ_RecieveCompleted);
IAsyncResult _result = _msgQ.BeginReceive();
_asyncList.Add(_result); // asyncList is a global variable of type System.Collections - > this allows the callback to remain open and therefore nit garbage collected while the async thread runs off on it's own
}
catch (Exception _ex)
{
throw new Exception("_msgQ_get Message threw the following error :- " + _ex);
}
}
//method to process message
public void _msgQ_RecieveCompleted(object sender, ReceiveCompletedEventArgs e)
{
try
{
//queue that have received a message
MessageQueue _mq = (MessageQueue)sender;
//get the messge off the queue
Message _mqmsg = _mq.EndReceive(e.AsyncResult);
//set the values back into a formatted struct
//now process your SQL....
Azure_SQL _azuresql = new Azure_SQL();
_azuresql.writeMessageToStorage((_TwitterStreamFeed)_mqmsg.Body);
//refresh queue just in case any changes occurred (optional)
_mq.Refresh();
//tell MessageQueue to receive next message when it arrives
_mq.BeginReceive();
}
catch (Exception _ex)
{
throw;
}
dll 2
public void writeMessageToStorage(_TwitterStreamFeed _msmq_message_as_TSF)
{
try
{
// now do something with the class - i..e write the values to the database
SqlConnection _azurecon = new SqlConnection(_AzuzeSQLConnection);
SqlCommand _sqlcmd = new SqlCommand();
//Setup the command string to call the stored procedure
//Add the parameter to the parameters collection of the command
blah blah blah......... Do SQL writing to Db
_azurecon.Open();
SqlDataReader _sqldr_tweet_place = _sqlcmd_place.ExecuteReader(CommandBehavior.CloseConnection);
}
//now close things off
_azurecon.Close();
}
catch (Exception _ex)
{
// Throw the error to preserve the original
throw;
}

The reason for this is that, internally, the MessageQueue class is explicitly swallowing the exception. Where the MessageQueue class raises the ReceiveCompleted event, it's inside of a try-catch statement - and the catch block is empty. Suffice it to say, if an exception occurs inside your ReceiveCompleted event handler, _msgQ_RecieveCompleted(), nothing's ever going to know it happened.
I see a couple of options, in order of preference.
Option 1 - Shift where the asynchronous call is made
Since this exception-swallowing behavior only occurs when using BeginReceive(), in MSMQ_GetMessage(), you can switch from using BeginReceive() to just Receive(). Then, make your call to MSMQ_GetMessage() asynchronous and any exception that gets thrown will be propagated as expected.
As a side note, a new(er) alternative for making asynchronous calls is available; the Task<> class. As opposed to the Thread class, Task<> has exception handling functionality built in. It does, however, require Framework 4 or higher. There is a good explanation of it's use described in the answer here.
Option 2 - Use a custom event
If refactoring the asynchronous call isn't an option, you can create a custom event in your class in 'dll 2' and subscribe to that event in 'Console.exe'. So when an exception occurs in _msgQ_RecieveCompleted(), you can raise the event and 'Console.exe' will be notified.

The MessageQueue.BeginReceive() method uses the standard .NET APM (Asynchronous Programming Model) pattern. It is very important to understand how it works to know how to properly deal with exceptions. Be sure to read the MSDN article, there are lots of other googable resources available.
In APM, the callback that tells you that a message was received in executed on a thread-pool thread. Which is a very efficient way to get code to run quickly. It is however also a very troublesome way when something goes wrong. The EndReceive() method call is likely to throw an exception, it does so to tell you that the receive operation could not be completed. A standard exception it will throw is ObjectDisposedException. Which will happen when the MessageQueue object gets disposed. In your case when your program terminates. You need to catch that exception and exit from your event handler, it is an expected exception and signals that nothing more useful is going to happen next since the queue was closed.
Then there's a raft of possible exceptions that can be raised by major mishaps in the message queue plumbing. Plus whatever you do with the message. Looks like you execute some Azure code, plenty of ways that can fall over. If you let such an exception escape from the callback method, like you do, then there's no catch clause anywhere in the call stack that is going to handle the exception. The standard way .NET deals with unhandled exceptions is to raise the AppDomain.UnhandledException event and terminate your program. If you didn't actually implement that event then there's nothing decent to look at to diagnose the reason your program ended, the Windows Error Reporting dialog has no good diagnostic.
Whether or not you should try to handle the exception and prevent the program from terminating is up to you. But it pretty strongly fits the "don't shoot the messenger" pattern, it is very unlikely your program can meaningfully continue to execute when such an exception is raised. It invariably takes a human to fix the problem, like restoring the network connection or fixing the message queue. If you do catch it then the odds that the same exception is raised over and over again is fairly likely. After all, there wasn't anything decent you could do in your code to repair the network.
So the best guidance here is to not try, just make sure that IT staff has a good diagnostic so they can repair the problem. Do implement the AppDomain.UnhandledException and display and log the e.UnhandledException.ToString() value. This will also let you learn the number of ways that your program can fail. There might be some conditions that are common enough to warrant catching, something like a temporary network outage. At that point you'll also know what to do about it, in other words what kind of code to write in the catch clause. There is no possible way you know what to write right now, you should therefore not try.
Last but not least, do note that you got yourself into this pickle because you used BeginReceive() unnecessarily. You've already got a perfectly good thread to do work on. But it doesn't do anything useful, it is stuck in the Console.ReadKey() method. Particularly in .NET 4.5 a very tricky method to call, it prevents other threads from writing anything to the console. So your error reporting won't work, it will deadlock when it tries to use Console.WriteLine() to write a diagnostic.
You might as well use MessageQueue.Read() instead. Now dealing with exceptions is a lot easier since they occur on the same thread. The MessageQueue.SynchronizingObject can also be helpful to get completion callbacks to occur on the main thread, but that only works in a GUI app, not in a console app.

Try catch not working properly

I am trying to insert a new member into database,
when there is a duplicate the catch block works ok.
But when there is a new member both try and catch blocks
return their messages. here is my code.
if (Request["cmd"] == "ins")
{
try{
mydb db = new mydb();
member newm = new member()
{
Id = Request["uid"],
Name = Request["uname"]
};
db.AddTomembers(newm);
db.SaveChanges();
Response.Write("ok");
Response.End();
}
catch(Exception s) {
Response.Write(s);
Response.End();
}
}

It's not really clear from the question, but I strongly suspect that the problem is that Response.End() throws an exception (ThreadAbortException) as documented:
To mimic the behavior of the End method in ASP, this method tries to raise a [ThreadAbortException] exception. If this attempt is successful, the calling thread will be aborted, which is detrimental to your site's performance. In that case, no code after the call to the End method is executed.
You're capturing that exception and then calling it again. Just move the Response.End to a finally block - or ideally, get rid of it entirely, structuring your control flow so you don't need it... again, as documented:
This method is provided only for compatibility with ASP—that is, for compatibility with COM-based Web-programming technology that preceded ASP.NET. If you want to jump ahead to the EndRequest event and send a response to the client, it is usually preferable to call CompleteRequest instead.
(Note that if you'd caught a specific exception rather than just Exception, you wouldn't have this problem either.)

If both ok and the exception message s are written, it's the Response.End() that is throwing the exception. Examine s to find out what is going on.
A guess: Is Response.End() considered harmful?

If you are trying to call the Response.End() on the Page_Load event of the asp.net, it will not work properly. Take a look at this thread: Response.End () doesn't abort current thread, and see how to solve it.
Try to avoid Responses on asp.net application, try to show panels, use labels to show messages.

Trying to understand exceptions in C#

I do not really use any try/catches in my code ever but I'm trying to break that habit and now get in to using exceptions.
I figure the most important place to have it in my application would be reading a file and I'm trying to implement that now but I'm unsure of the "best-practices" for doing so. Currently I'm doing something like this:
private void Parse(XDocument xReader)
{
IEnumerable<XElement> person = xReader.Descendants("Person").Elements();
foreach (XElement e in person)
personDic[e.Name.ToString()] = e.Value;
if (personDic["Name"] == null || personDic["Job"] == null || personDic["HairColor"] == null)
throw new KeyNotFoundException("Person element not found.");
}
But I am unsure if this is correct. I have this for handling it:
try
{
personsReader.Read(filename, persons);
}
catch (KeyNotFoundException e)
{
MessageBox.Show(e.Message);
return;
}
// Do stuff after reading in the file..
However when showing e.Message it just shows the generic KeyNotFoundException error message and not by custom error message. Also I'm not sure if in general I am going about this whole "exception handling stuff" properly. I do return in the catch because if the file is not read successfully obviously I just want to pretend like the user never tried to open a file and let him try again with another file.
Am I doing this properly? Again I am fairly new to using exceptions and I want to make it sure I got it down right before continuing on and applying this to the rest of my program.
Also, why do people say not to do catch (Exception e)? It seems like in this case I would want to do that because regardless of what error occurs when reading in a file, if there is an error, I want to stop reading the file, display the error message, and return. Wouldn't that always be the case? I can understand not wanting to handle Exception e if you would want to handle something differently based on the exception but in this case wouldn't I want to just handle the base exception class in case anything goes wrong?

You should catch exceptions when you can handle the condition and do something useful. Otherwise you should let it bubble up the call stack and perhaps someone above you can handle it. Some apps have unhandled exception handlers to handle it at the outer most layer but in general, unless you know you have some useful way to handle it, let it go.
In your case, you're handling not being able to read a resource and informing the user. You're handling it. Concerning the generic exception, one thing you can do is catch and re-throw a better exception. If you do that, make sure you incude the root cause exception as the inner exception. You can also trace or log the details if appropriate.
throw new MyGoodExceptionType ("Could not read file", e); // e is caught inner root cause.
Now the UI shows a good error and perhaps the inner root cause is in a log etc...
Some typical mistakes:
Handling exceptions deep in the stack in a generic library method: Remember that a common library function may get called in many different code paths. You likely don't have the context whether it should be handled and whether it's appropriate to handle it. the caller higher in the stack likely has context and knows whether it's safe to handle. Typically that means higher layers of code decide to handle. In lower layers, typically you let them flow.
Swallowing Exception: Some code catches exceptions (especially lower in the stack) and then the root condition just evaporates making it maddening to debug. Once agan, if you can handle it, do so. If not, let it go.
Exceptions should be exceptional: Don't use excpetions for flow control. For example, if you're reading a resource, don't try and read and then catch the exception and make a decision point. Instead, call ifexists, check bool and make decisions in your code. this especially helps when you set the debugger to break on exceptions. You should be able to run clean and if the debugger breaks, it should be a real issue. Having the debugger break constantly when debugging is problematic. I personally like throwing exceptions very rarely and always try to avoid for flow control.
Hope that helps.

Ok, first...
... This iss not KeynotFoundException, it should be ArgumentException.... the Argument provided is not valid.
The documentation clearly states:
The exception that is thrown when the key specified for accessing an element in a collection does
not match any key in the collection.
Compare that with:
The exception that is thrown when one of the arguments provided to a method is not valid
Now:
Also, why do people say not to do catch (Exception e)?
Becasue this swallows the exception and makes it impossible to have central error handling / logging in place. ONLY handle exception you expect, UNLESS it is a catch / close something or log it / rethrow (i.e. throw;). Then have a central appdomain handler that gets every uncaptured exceptin and logs it ;) It can not handle anything - because exceptions at that level are unexpected,. It should basically write the excption to a file and be done, possibly with a UI (application has t obe restartet).

As far as what you're doing, it looks mostly ok. I can't speak to whether or not you should be throwing exceptions at that particular point, but the throwing and catching is correctly done. As far as the message, it should be working the way it stands. Try displaying e.ToString() to see the call stack. It could be that simply doing person["Name"] is throwing the KeyNotFoundException first.
As to the question of catching Exception, it's not always bad. Sometimes you cannot predict all possible exceptions, and it's sometimes a good thing to handle any possible failure. However, it gives you no way to handle particular exceptions differently.
As an example, if you get KeyNotFoundException, you might mention something about how the file is incorrectly formatted, and maybe display the file to the user on the screen. If you get FileNotfoundException, you could show them the path and open up an OpenFileDialog to have them select a new file. Exceptions due to security permissions you could display instructions to have them elevate your permissions. Some exceptions may even be recoverable (perhaps one element is badly formatted, but the rest are ok; should it fail the whole thing?)
But, it's ok to catch everything if that's how you want to design it. The most solid program out there would catch every possible exception and handle it in very specific ways, instead of presenting the raw exception to the user. It makes for a better user experience, and gives you ways to work around the problems that can happen.

Most of the time you may not care about the type of exception you get so catching the generic Exception is fine, however there are specific situations in which you would actually want to catch the relevant exception (not just the generic Exception).
One particular example is if you have a thread and you want to interrupt it from a blocking call, in that case you have to distinguish between the InterruptException and the Exception.
Consider this example: you have a thread which runs the Read every minute for 5 minutes (it's not a very realistic example, but it should give you an idea of why you want to handle different exceptions). You have to stop the thread after 5 minutes because your application is going to shut down and you don't want to wait another minute for the running flag to be read... after all, you don't want your user to be waiting for an entire minute just to shut down the application. In order to stop the thread right away, you set the flag to false and you call Interrupt on your thread. In this case you specifically have to catch the ThreadInterrupted exception, because it tells you that you should exit the loop. If you catch another exception, then you've failed to perform the task, but you don't want to give up on the job all together and you would like to try and read again the next minute. This depicts how your requirements dictate the type of exceptions you need to handle. Here is the example in code:
bool running = true;
Thread t = new Thread(()=>
{
while(running)
{
try
{
// Block for 1 minute
Thread.Sleep(60*1000);
// Perform one read per minute
personsReader.Read(filename, persons);
}
catch (KeyNotFoundException e)
{
// Perform a specific exception handling when the key is not found
// but do not exit the thread since this is not a fatal exception
MessageBox.Show(e.Message);
}
catch(InterruptException)
{
// Eat the interrupt exception and exit the thread, because the user
// has signalled that the thread should be interrupted.
return;
}
catch(Exception e)
{
// Perform a genetic exception handling when another exception occurs
// but do not exit the thread since this is not a fatal error.
MessageBox.Show("A generic message exception: " + e.Message);
}
}
});
t.IsBackground = true;
t.Start();
// Let the thread run for 5 minutes
Thread.Sleep(60*5000);
running = false;
// Interrupt the thread
t.Interrupt();
// Wait for the thread to exit
t.Join();
Now on to your other problem with the exception not showing up: note that you're accessing person[e.Name.ToString()] = e.Value which requires a key lookup and if the key is not in the map, then you may get a KeyNotFoundException. That would be the generic exception that you're catching and your custom exception will never be thrown because person[e.Name.ToString()] may throw before you even get to your code.
foreach (XElement e in person)
person[e.Name.ToString()] = e.Value; // <-- May be throwing the KeyNotFoundException
if (person["Name"] == null || person["Job"] == null || person["HairColor"] == null)
throw new KeyNotFoundException("Person element not found.");
Furthermore, you don't want to throw a KeyNotFoundException when you've actually found the key but you didn't find a corresponding value: if person["Name"] == null evaluates to true, then the key "Name" was actually found in the person dictionary, so throwing the KeyNotFoundException would be misleading to anybody who catches that exception. In the case that your value is null, then it would probably not be advisable to throw an exception anyway... it really isn't an exceptional case. You could return a flag indicating that the key was not found:
public bool PerformRead(/*... parameters ...*/)
{
foreach (XElement e in person)
{
// Avoid getting the KeyNotFoundException
if(!person.ContainsKey(e.Name.ToString()))
{
person.Add(e.Name.ToString(), "some default value");
}
person[e.Name.ToString()] = e.Value;
}
if (person["Name"] == null || person["Job"] == null || person["HairColor"] == null)
{
return false;
}
else
{
return true;
}
}

I'm not quite sure why you're not getting your custom error message. That should be happening (unless it's something else throwing a KeyNotFoundException, not the one that you're explicitly throwing).
Also, generally you should put all the code that relies on the file read being successful inside the try, which is often the rest of the body of your method. You no longer would need a return inside your catch block, and subsequent code that doesn't rely on the file read being successful could still execute after a failure.
Example:
public static void Main()
{
var filename = "whatever";
try
{
personsReader.Read(filename, persons);
var result = personsReader.DoSomethingAfterReading();
result.DoSomethingElse();
}
catch (KeyNotFoundException e)
{
MessageBox.Show(e.Message);
}
finally
{
personsReader.CloseIfYouNeedTo();
}
DoSomeUnrelatedCodeHere();
}
And the reason it's good practice not to catch any old Exception e is because you only want to catch and handle the exceptions you're expecting to get. If you get a different kind of exception that you weren't expecting to get, typically this means that something novel failed in a way you didn't anticipate, and you want this behavior to be noticeable, not just get swept under the rug with all the regular error-handling code.
A lot of production-level systems will have one big try/catch around the entire program that catches any exception and performs logging and cleanup before crashing gracefully. This is complemented by having specific try/catch blocks deeper inside the code that handle expected exceptions in a well-defined manner. For unexpected exceptions, you could always just let the CLR bomb ungracefully and figure out what happened from that.
Here's an example of a novel exception. What if something goes terribly wrong and in this line:
IEnumerable<XElement> person = xReader.Descendants("Person").Elements();
...you get an OutOfMemoryException? Should you really just display a popup to the user and allow your program to try to carry on like normal, even though there's simply no way it will be able to? And what if, because you failed silently on an OutOfMemoryException, you later attempt to dereference a null reference, and get a NullReferenceException that causes your program to crash? You'll have a devil of a time trying to track down the root cause of why that reference was null.
Best way to suss out a bug is to fail fast and fail noisily.

"Exceptions are for Exceptional circumstances" - unknown
Don't use the to essentially pass a message out of a method to the calling method. Always try to gracefully handle things. When something really odd happens then throw an exception. This is something that dev not real familiar with how to use exceptions do a lot.

In your code you are triggering the xxx when you evaluate the condition inside the if statement.
Asking person["Name"] == null || person["Job"] == null || person["HairColor"] == null will fail if any of those keys are not in your dictionary.
You need to do this instead:
if (!person.ContainsKey("Name"] ||
!person.ContainsKey("Job"] ||
!person.ContainsKey("HairColor"))
So, your call to throw the exception is never executed! And that's why you never see your message.
I would keep your habit of not doing exceptions for this kind of coding. Exceptions are expensive and can cause real issues in your code to be hidden.
Don't catch general exceptions and don't create exceptions for non-exceptional circumstances.