We Keep Coding

why error message "Operation failed after 3 times" appears even there is no re-try for error code 100

I want to re-try execution of code block if error code = 100 and if all re-try fails I want to print the message "Operation failed after 3 times" for this case only.
I have below Polly policy defined,
var retryPolicy = Policy
.Handle<Exception>(ex =>
{
var errorMessage = ex.InnerException?.Message;
if (errorMessage == null) return false;
try
{
dynamic error = JsonConvert.DeserializeObject(errorMessage);
if (error != null && error.errorCode == 100)
{
Console.WriteLine("Try to run again...");
return true;
}
Console.WriteLine("Error occurred: " + ex.Message);
return false;
}
catch (Exception)
{
Console.WriteLine("Exception Error occurred: " + ex.Message);
return false;
}
})
.WaitAndRetryAsync(2, _ => TimeSpan.FromSeconds(10));
And below is the code for policy execution,
try
{
var myClass = new MyClass();
await retryPolicy.ExecuteAsync(async () => await myClass.Get());
}
catch (Exception)
{
Console.WriteLine("Operation failed after 3 times");
}
For below code everything perfect and I am getting desired result as well,
Try to run again...
Try to run again...
Try to run again...
Operation failed after 3 times
public async Task Get()
{
await Task.Delay(1);
throw new Exception("message", new Exception("{\"message\":\"inner exception\",\"errorCode\":\"100\"}"));
}
But when I am execution below code ( no error code = 100), then my re-try not happening but the message "Operation failed after 3 times" is also printing in console. What's the reason for it and how to avoid it?
Exception Error occurred: message
Operation failed after 3 times
public async Task Get()
{
await Task.Delay(1);
throw new Exception("message", new Exception("hey you"));
}

In general it's not good, to have a try-catch-block in an Exception handling. In your case it's generated because of your JsonConvert, but do you really need that?
I would suggest, to throw an Exception of type Win32Exception, because here you can define an error code. Like this
throw new Exception("message", new System.ComponentModel.Win32Exception(100));
With that you don't need to do a Json-Deserialzation and you got no "second" try-catch-block.
"Operation failed after 3 times" is printed on every exception, says also on exceptions without an inner one or error code == 100. To fix that, you can now do something like
try
{
var myClass = new MyClass();
await retryPolicy.ExecuteAsync(async () => await myClass.Get());
}
catch (Exception ex)
{
bool innerIsWin32 = ex.InnerException is System.ComponentModel.Win32Exception;
if (innerIsWin32)
{
var w32ex = (Win32Exception)ex.InnerException
if (w32ex.ErrorCode == 100)
{
Console.WriteLine("Operation failed after 3 times");
}
}
}
Starting with C# 6, when can be used in a catch statement to specify a condition that must be true for the handler for a specific exception to execute.
catch (Win32Exception ex) when (ex.InnerException is Win32Exception) {
var w32ex = (Win32Exception)ex.InnerException;
var code = w32ex.ErrorCode;
if (code == 100)
{
Console.WriteLine("Operation failed after 3 times");
}
}

Let me present here a solution which shows how should you solve this problem with Polly.
Retries vs Attempts
Let's start with clarifying some terminology.
In your code sample you have set the retryCount to 2 but you have 3 Try to run again... messages on your Console. The reason behind this is that in total you had 3 attempts: 1 initial attempt and 2 retries.
Because you have put your logging inside the exceptionPredicate that's why it is evaluated three times:
After the initial attempt
After the first retry attempt
After the second retry attempt
The last one is bit odd since it does not trigger a retry. Why? Because you would exceed the retry count with that.
Later in this post we will discuss where should you put the logging.
The exceptionPredicate
Please try to keep this predicate as simple as possible. As you have seen it it is evaluated after each attempt (not after each retry)!
Rather than having a try-catch inside this predicate you can instruct the Json.Net deserializer to silently ignore errors if it can not parse the input as json.
var silentlyIgnoreError = new JsonSerializerSettings
{
Error = (_, args) => args.ErrorContext.Handled = true
};
With this settings your predicate could be streamlined like this
.Handle<Exception>(ex =>
{
var errorMessage = ex.InnerException?.Message;
if (errorMessage == null)
return false;
if (JsonConvert.DeserializeObject(errorMessage, silentlyIgnoreError) == null)
return false;
var errorCode = (string?)JObject.Parse(errorMessage)["errorCode"];
return int.TryParse(errorCode, out int errorNumber) && errorNumber == 100;
})
If the exception does not contain an inner then do not retry
If the inner excepion's message can't be parsed as json then do not retry
If the json does not contain an errorCode field then do not retry
If the json contains an errorCode field but the value is not an integer then do not retry
If the json contains an errorCode field with an integer value but it's different than 100 then do not retry
Otherwise do retry :D
As you can see there is no logging here.
Logging
The logging logic should be placed inside the onRetry/onRetryAsync delegate which is executed when the policy has already decided that is should be triggered but before the sleep.
.WaitAndRetryAsync(2,
_ => TimeSpan.FromSeconds(10),
(ex, _, ctx) =>
{
Console.WriteLine($"Try to run again... ({ex.Message})");
ctx.IncreaseRetryCount();
});
With this setup you would see only two Try to run again... lines
One after the initial attempt
One after the first retry attempt
I've used a special overload of the onRetry which has access to the Context
It gives us ability to store information between retry attempts
It also allows us to access that information after the policy execution
The usage of Context
I've defined two extension methods to ease the usage of the Context, which is a Dictionary<string, object> under the hood
public static class ContextExtensions
{
private static readonly string key = "RetryCount";
public static void IncreaseRetryCount(this Context context)
{
var retryCount = GetRetryCount(context);
context[key] = ++retryCount;
}
public static int GetRetryCount(this Context context)
{
context.TryGetValue(key, out object count);
return count != null ? (int)count : 0;
}
}
The IncreaseRetryCount is called whenever a retry will be triggered
The GetRetryCount is called after the execution of the policy
The execution of the policy
You can execute the policy not just with the Execute/ExecuteAsync but with the ExecuteAndCapture/ExecuteAndCaptureAsync as well.
It returns a PolicyResult/PolicyResult<T> object which has the following properties:
Outcome: Whether the policy/chain of policies succeeded or failed
FinalException: In case of failure the final exception
Context: That Context object which was used during the execution
(Result: If the policy had been defined in a way that it should return something)
As you might expect in case of a Failure it won't throw an exception.
If you would use ExecuteAndCaptureAsync then your code would look like this:
var result = await retryPolicy.ExecuteAndCaptureAsync(async () => await Get());
Console.WriteLine($"Operation has failed after the initial attempt + {result.Context.GetRetryCount()} retry attempt(s)");
For the sake of completeness here is the full source code
var silentlyIgnoreError = new JsonSerializerSettings { Error = (_, args) => args.ErrorContext.Handled = true };
var retryPolicy = Policy
.Handle<Exception>(ex =>
{
var errorMessage = ex.InnerException?.Message;
if (errorMessage == null) return false;
if (JsonConvert.DeserializeObject(errorMessage, silentlyIgnoreError) == null) return false;
var errorCode = (string?)JObject.Parse(errorMessage)["errorCode"];
return int.TryParse(errorCode, out int errorNumber) && errorNumber == 100;
})
.WaitAndRetryAsync(2,
_ => TimeSpan.FromSeconds(10),
(ex, _, ctx) =>
{
Console.WriteLine($"Try to run again... ({ex.Message})");
ctx.IncreaseRetryCount();
});
var result = await retryPolicy.ExecuteAndCaptureAsync(async () => await Get());
Console.WriteLine($"Operation has failed after the initial attempt + {result.Context.GetRetryCount()} retry attempts");
public static class ContextExtensions
{
private static readonly string key = "RetryCount";
public static void IncreaseRetryCount(this Context context)
{
var retryCount = GetRetryCount(context);
context[key] = ++retryCount;
}
public static int GetRetryCount(this Context context)
{
context.TryGetValue(key, out object count);
return count != null ? (int)count : 0;
}
}
UPDATE #1: Some correction
This piece of code:
var result = await retryPolicy.ExecuteAndCaptureAsync(async () => await Get());
Console.WriteLine($"Operation has failed after the initial attempt + {result.Context.GetRetryCount()} retry attempts");
assumed that the operation would always fail.
The correct way of handling this should be written like this:
var result = await retryPolicy.ExecuteAndCaptureAsync(async () => await Get());
var outcome = result.Outcome == OutcomeType.Successful ? "completed" : "failed";
Console.WriteLine($"Operation has {outcome} after the initial attempt + {result.Context.GetRetryCount()} retry attempts");
The OutcomeType can be either Successful or Failure.

Exception handling on console application async code

Hoping that someone will be able to point me in the right direction. I have a console application that uses async method calls and I'm having a bit of trouble with the try/catch error blocks. Basically, when a catch is triggered it executes the logging code within the catch but it still throws the error up to the top level and terminates the overall execution of the app.
I've been looking through other queries about this and from what I've seen this type of behaviour can be caused by either an async void or an async Task with no await. Looking at my code I don't think I have either so I'm a bit stumped.
The error being introduced is a missing connection string (in the context file, not shown), which should throw a handled exception in the PayloadRepo method. Stepping through the debugger this error is indeed caught and logged as expected, however it still seems to bubble back up to the Main method and cause the overall execution to break.
Can anyone spot what may be happening?
Thank you!
static async Task Main(string[] args)
{
BuildLogConfiguration();
BuildDependencyContainer();
await RunProgram();
}
static async Task RunProgram()
{
systemLog.Info("Started full process at: " + System.DateTime.Now);
using (var scope = Container.BeginLifetimeScope())
{
var payloadService = scope.Resolve<IPayloadService>();
await payloadService.ProcessPayloadData(scope);
}
systemLog.Info("Completing full process at: " + System.DateTime.Now);
}
public class PayloadService : IPayloadService
{
private static readonly ILog _systemLog = LogManager.GetLogger(MethodBase.GetCurrentMethod().DeclaringType);
public async Task ProcessPayloadData(ILifetimeScope scope)
{
await PayloadData_Process(scope);
}
private async Task PayloadData_Process(ILifetimeScope scope)
{
var repo = scope.Resolve<IPayloadRepo>();
var payloadList = await repo.Payload_Get();
}
}
public class PayloadRepo : IPayloadRepo
{
public async Task<IEnumerable<Payload>> Payload_Get()
{
using (var context = new ASGB_DataLayerContext())
{
try
{
var payloadList = context.Payloads;
var result = await payloadList.ToListAsync();
return result;
}
catch (Exception ex) <<--- CATCHES THIS AND EXECUTES CODE WITHIN, BUT IT STILL THROWS AN ERROR ON RunProgram().Wait() AND TERMINATES PROGRAM
{
systemLog.Error(ex.Message);
systemLog.Error(ex.InnerException);
var entries = context.ChangeTracker.Entries().Where(e => e.State != EntityState.Unchanged);
foreach (var entry in entries)
{
foreach (var prop in entry.CurrentValues.Properties)
{
var val = prop.PropertyInfo.GetValue(entry.Entity);
systemLog.Error($"{prop.ToString()} ~ ({val?.ToString().Length})({val})");
}
}
systemLog.Error("------------------------------------------------------------------------");
return null;
}
}
}
}

Calling webservice in parallel using aysnc

I'm trying to write a winforms app that retrieves a list of ids from a SOAP web service, then goes through that list, calling another web service to get individual details. To speed things up I'm doing these requests in parallel, in batches of 15.
When the web service calls all return OK, my application works great. However in my testing, if 2 or more requests timeout (I'm monitoring the requests using Fiddler and see I'm getting a 408), then it appears that the process hangs and I never get the completion message my programs should be displaying.
What I'd like to do if possible is retry these requests, or just tell the user to press the button again because an error occurred.
When adding the service reference, I made sure the generate task-based operations option was used.
Here is the code I've got so far (with some details renamed as not to give away the company I'm interfacing with):
private async void btnDownload_Click(object sender, EventArgs e)
{
try
{
Stopwatch watch = new Stopwatch();
watch.Start();
IProgress<string> progress = new Progress<string>(s =>
{
lbinfo.Items.Insert(0, s);
});
await GetData(progress);
watch.Stop();
progress.Report("Took " + (watch.ElapsedMilliseconds / 1000f) + " seconds. All done.");
}
catch(Exception ex)
{
MessageBox.Show("Main Click: "+ex.ToString());
}
}
private async Task GetData(IProgress<string> progress)
{
try
{
DownloadSoapClient client = new DownloadSoapClient();
client.Open();
progress.Report("Getting master list.");
List<int> uniqueIds = await GetMasterList(client); //this always seems to work
progress.Report("Downloaded master list. Found "+uniqueIds.Count +" unique ids.");
var detailedData = await GetIdDataRaw(uniqueIds, client,progress);
client.Close();
}
catch (Exception ex)
{
progress.Report("GetData: "+ex);
}
}
private async Task<List<DownloadResponse>> GetIdDataRaw(List<int> ids, DownloadSoapClient client, IProgress<string> progress)
{
using (var throttler = new SemaphoreSlim(15))
{
var allTasks = ids.Select(async x =>
{
await throttler.WaitAsync();
try
{
progress.Report(x.ToString());
return await client.DownloadAsync(username, password, x);
}
catch (Exception ex)
{
progress.Report("Error getting id:" + x + " " + ex.Message);
return null;
}
finally
{
throttler.Release();
}
});
return (await Task.WhenAll(allTasks)).ToList();
}
}
Typically the master list is around 1000 entries, and when only 1 times out, as expected I get a message saying there was an error getting id xxxx, and then "Took xx seconds. All done.".
So far I've tried a few other things, such as creating a client for each request, and refactoring to use Task.WhenAny in a while loop, but these exhibit the same behaviour when 2 or more requests fail.

Best way to handle exception in multi task to call WCF operation

I have implemented a service name ExamClient which have two operations one is Ping which return a basic string which mean service is available and one is FindStudy which search in DB it may take a long to be proceeded.
In the other side I have several endpoints of ExamClient I wand to run FindStudy per end point by task so in a Dispatcher I have something like this:
public FindStudies_DTO_OUT FindStudies(FindStudies_DTO_IN findStudies_DTO_IN)
{
List<Study_C> ret = new List<Study_C>();
List<Task> tasks = new List<Task>();
foreach (var sp in Cluster)
{
string serviceAddress = sp.GetLibraryAddress(ServiceLibrary_C.PCM) + "/Exam.svc";
var task = Task.Run(() =>
{
ExamClient examClient = new ExamClient(serviceAddress.GetBinding(), new EndpointAddress(serviceAddress), Token);
var ping = Task.Run(() =>
{
examClient.Ping();
});
if (!ping.Wait(examClient.Endpoint.Binding.OpenTimeout))
{
Logging.Log(LoggingMode.Warning, "Timeout on FindStudies for:{0}, address:{1}", sp.Name, serviceAddress);
return new List<Study_C>(); // if return null then need to manage it on ret.AddRange(t.Result);
}
return (examClient.FindStudies(findStudies_DTO_IN).Studies.Select(x =>
{
x.StudyInstanceUID = string.Format("{0}|{1}", sp.Name, x.StudyInstanceUID);
x.InstitutionName = sp.Name;
return x;
}));
});
task.ContinueWith(t =>
{
lock (ret)
{
ret.AddRange(t.Result);
}
}, TaskContinuationOptions.OnlyOnRanToCompletion);
task.ContinueWith(t =>
{
Logging.Log(LoggingMode.Error, "FindStudies failed for :{0}, address:{1}, EXP:{2}", sp.Name, serviceAddress, t.Exception.ToString());
}, TaskContinuationOptions.OnlyOnFaulted);
tasks.Add(task);
}
try
{
Task.WaitAll(tasks.ToArray());
}
catch (AggregateException aggEx)
{
foreach (Exception exp in aggEx.InnerExceptions)
{
Logging.Log(LoggingMode.Error, "Error while FindStudies EXP:{0}", exp.ToString());
}
}
return new FindStudies_DTO_OUT(ret.Sort(findStudies_DTO_IN.SortColumnName, findStudies_DTO_IN.SortOrderBy));
}
First I have to run Ping per end point to know connection is established
after that FindStudy.
if there are three end pints in Cluster six task be run in parallel mode, 3 for Ping and 3 for FindStudy.
I think something is wrong with my code to handle exception nice...
So what is the best way to implement this scenario ?
thanks in advance.

Let me throw my answer to simplify and remove unnecessary code block. And bit of explanation along the code.
public FindStudies_DTO_OUT FindStudies(FindStudies_DTO_IN findStudies_DTO_IN)
{
// Thread-safe collection
var ret = new ConcurrentBag<Study_C>()
// Loop cluster list and process each item in parallel and wait all process to finish. This handle the parallism better than task run
Parallel.Foreach(Cluster, (sp) =>
{
var serviceAddress = sp.GetLibraryAddress(ServiceLibrary_C.PCM) + "/Exam.svc";
ExamClient examClient = new ExamClient(serviceAddress.GetBinding(), new EndpointAddress(serviceAddress), Token);
try
{
examClient.Ping();
// declare result variable type outside try catch to be visible below
var result = examClient.FindStudies(findStudies_DTO_IN);
}
catch(TimeoutException timeoutEx)
{
// abort examclient to dispose channel properly
Logging.Log(LoggingMode.Warning, "Timeout on FindStudies for:{0}, address:{1}", sp.Name, serviceAddress);
}
catch(FaultException fault)
{
Logging.Log(LoggingMode.Error, "FindStudies failed for :{0}, address:{1}, EXP:{2}", sp.Name, serviceAddress, fault.Exception.ToString());
}
catch(Exception ex)
{
// anything else
}
// add exception type as needed for proper logging
// use inverted if to reduce nested condition
if( result == null )
return null;
var study_c = result.Studies.Select(x =>
{
x.StudyInstanceUID = string.Format("{0}|{1}", sp.Name, x.StudyInstanceUID);
x.InstitutionName = sp.Name;
return x;
}));
// Thread-safe collection
ret.AddRange(study_c);
});
// for sorting i guess concurrentBag has orderby; if not working convert to list
return new FindStudies_DTO_OUT(ret.Sort(findStudies_DTO_IN.SortColumnName, findStudies_DTO_IN.SortOrderBy));
}
Note : Code haven't tested but the gist is there. Also I feels like task.run inside task.run is bad idea can't remember which article I read it (probably from Stephen Cleary not sure).

ContinueWith not waiting for task to complete

I have a function (below) that I retrieve data from an API. If I set a breakpoint at the line that deserializes it, then I can see that it is populated with data which is great.
When I continue on, it goes into the second function (below) and it throws an error. The error says next to it Not yet computed, and therefore throwing an exception.
When I do it with a small list it works just fine (I presume its cos it's a small set of data).
How is this possible when I'm using ContinueWith (waiting for the task to complete)?
public static async Task<Data> GetAllCardsInSet(string setName)
{
setName = WebUtility.UrlEncode(setName);
var correctUri = Path.Combine(ApiConstants.YugiohGetAllCardsInSet, setName);
Console.WriteLine();
using (var httpClient = new HttpClient())
{
var response =
await httpClient.GetAsync(correctUri);
var result = await response.Content.ReadAsStringAsync();
var cardData = JsonConvert.DeserializeObject<CardSetCards>(result);
for (int i = 0; i < cardData.Data.Cards.Count; i++)
{
cardData.Data.Cards[i] = FormatWords(cardData.Data.Cards[i]);
}
return cardData.Data;
}
}
private void GetYugiohCardsAndNavigate(string name)
{
var cardSetData = YugiohRequester.GetAllCardsInSet(_selectedCardSet.Name).ContinueWith((result) =>
{
//var cards = await YugiohRequester.GetAllCardsInSet(_selectedCardSet.Name);
try
{
this.mainPage.NavigateToYugiohCardListPage(result.Result);
}
catch (Exception e)
{
HelperFunctions.ShowToastNotification("Trading Card App", "Sorry, we could not fetch this set");
}
});
}

Your GetAllCardsInSet method no need to change.
But using of this method can be refactored.
Method GetAllCardsInSet return Task and you not observed the completion of the this Task.
You need to check is Task completes succesfully, easiest approach to use await keyword. Awaiting task will unwrapp returned value or throw exception if task completed with exception.
For using async/await in the GetYugiohCardsAndNavigate change method signature to aynchronous and returning Task
private async Task GetYugiohCardsAndNavigate(string name)
{
try
{
var cardSetData = await YugiohRequester.GetAllCardsInSet(_selectedCardSet.Name);
this.mainPage.NavigateToYugiohCardListPage(cardSetData);
}
catch (Exception e)
{
HelperFunctions.ShowToastNotification("Trading Card App",
"Sorry, we could not fetch this set");
}
}

you called an async method in a sync method without Wait. It should have been done like:
YugiohRequester.GetAllCardsInSet(_selectedCardSet.Name).ContinueWith((result) =>
{
//var cards = await YugiohRequester.GetAllCardsInSet(_selectedCardSet.Name);
try
{
this.mainPage.NavigateToYugiohCardListPage(result.Result);
}
catch (Exception e)
{
HelperFunctions.ShowToastNotification("Trading Card App", "Sorry, we could not fetch this set");
}
}).Wait();