We Keep Coding

Odd behaviour when in async await (maybe because of multiple requests)

I have the following use-case, FetchDataService (injected as scoped), is fetching data from API using httpClient.
Then the service analyze the data and run await SaveAsync it to the database.
Everything till this point works great.
Then I added another service: AddressEnrichmentService (injected as scoped) that during the "analyze" phase is calling external service asyncly and bring more data.
This service is messing the entire app. I get random exceptions from the database handler (NHibernate) that is indicating some thread problems. I really can't put my finger on it. few different random exceptions...
These exceptions are thrown when running await SaveAsync. (I will add them in the end).
public async Task Fetch(string url, UserModel userModel ){
string res = null;
using (var httpClient = new HttpClient()){
res = await httpClient.GetStringAsync(url);
}
await databaseLogRepository.SaveAsync(new Log{Message = "The data is here..." + res});
var profile = JsonConvert.DeserializeObject<ProfileModel >(res);
profile.Data.Locations.ForEach(async x =>
{
var address = await addressEnrichmentService.EnrichAsync(x.StreetAddress + " " + x.Locality + " " + x.Country);
userModel.Address = address;
await databaseLogRepository.SaveAsync(new Log{Message = "enriched address"});
});
}
The problematic service:
public class AddressEnrichmentService : IAddressEnrichmentService
{
public async Task<AddressModel> EnrichAsync(string address)
{
string res = null;
using (HttpClient client = new HttpClient())
{
var url = "https://maps.googleapis.com/maps/api/geocode/json?key=__KEY__&address=" + HttpUtility.UrlEncode(address);
res = await client.GetStringAsync(url);// probably the problematic row
}
return JsonConvert.DeserializeObject<AddressModel>(res);
}
}
Usage
UserModel userModel = new UserModel();
await fetchDataService.Fetch(url, userModel);
await userRepo.SaveAsync(userModel);
Again addressEnrichmentService.EnrichAsync is messing everything up.
How do I know EnrichAsync is messing it up?
if I convert res = await client.GetStringAsync(url); to res = client.GetStringAsync(url).GetAwaiter().GetResult(); I dont get any error. 100 times of 100 trys I get no error. if I roll it back to res = await client.GetStringAsync(url); I get an error every time.
About the exceptions, again, im getting some indications that the error related to concurrency, this is the save method:
public async Task<T> SaveAsync(T entity)
{
using (var tr = session.BeginTransaction())
{
//session is injected as scoped
await session.SaveOrUpdateAsync(entity);
tr.Commit();
}
return entity;
}
MySqlException: There is already an open DataReader associated with this Connection which must be closed first.
NHibernate.HibernateException: 'Flush during cascade is dangerous'

Here is the problem:
profile.Data.Locations.ForEach(async x =>
The ForEach method accepts an Action, not an asynchronous delegate. So your async delegate does not produce an awaitable Task. It produces a void. The result is that you end up with an async void delegate, that has tortured myriads of developers before you (search for async void to see a never-ending list of related questions). Since the started asynchronous operations are not awaited (they are not awaitable), they are running concurrently. Your database is receiving multiple concurrent requests, and obviously can't handle them well. Not to mention that if any of these operations fail, the error will be thrown on the captured synchronization context, causing most likely the process to crash. You can't handle the error of an async void invocation, by wrapping the invocation in a try/catch block.
To solve the problem, just replace the fancy ForEach with a plain vanilla foreach. Each Task will be properly awaited, the unwanted parallelism will not occur, and the problem will be solved.
foreach (var x in profile.Data.Locations)
{
var address = await addressEnrichmentService.EnrichAsync(
x.StreetAddress + " " + x.Locality + " " + x.Country);
userModel.Address = address;
await databaseLogRepository.SaveAsync(new Log{Message = "enriched address"});
}

azure functions running in sequence, parallel desired

I have an azure function that I'm calling in parallel using postasync...
I arrange all my tasks in a queue and then wait for the responses in parallel using "WhenAll".
I can confirm that there is a burst of HTTP activity out to Azure and then HTTP activity stops on my local machine while I wait for responses from Azure.
When I monitor the function in Azure Portal, it looks like the requests are arriving every three seconds or so, even though from my side there is no network traffic after the initial burst.
When I get my results back, they are arriving in sequence, in the exact same order I sent them out, even though the Azure Portal monitor indicates that some functions take 10 seconds to run and some take 3 seconds to run.
I am using Azure functions Version 1 with a consumption service plan.
CentralUSPlan (Consumption: 0 Small)
My host.json file is empty ==> {}
Why is this happening? Is there some setting that is required to get azure functions to execute in parallel?
public async Task<List<MyAnalysisObject>> DoMyAnalysisObjectsHttpRequestsAsync(List<MyAnalysisObject> myAnalysisObjectList)
{
List<MyAnalysisObject> evaluatedObjects = new List<MyAnalysisObject>();
using (var client = new HttpClient())
{
var tasks = new List<Task<MyAnalysisObject>>();
foreach (var myAnalysisObject in myAnalysisObjectList)
{
tasks.Add(DoMyAnalysisObjectHttpRequestAsync(client, myAnalysisObject));
}
var evaluatedObjectsArray = await Task.WhenAll(tasks);
evaluatedObjects.AddRange(evaluatedObjectsArray);
}
return evaluatedObjects;
}
public async Task<MyAnalysisObject> DoMyAnalysisObjectHttpRequestAsync(HttpClient client, MyAnalysisObject myAnalysisObject)
{
string requestJson = JsonConvert.SerializeObject(myAnalysisObject);
Console.WriteLine("Doing post-async:" + myAnalysisObject.Identifier);
var response = await client.PostAsync(
"https://myfunctionapp.azurewebsites.net/api/BuildMyAnalysisObject?code=XXX",
new StringContent(requestJson, Encoding.UTF8, "application/json")
);
Console.WriteLine("Finished post-async:" + myAnalysisObject.Identifier);
var result = await response.Content.ReadAsStringAsync();
Console.WriteLine("Got result:" + myAnalysisObject.Identifier);
return JsonConvert.DeserializeObject<MyAnalysisObject>(result);
}

Azure ML web service times out

I have created a simple experiment in Azure ML and trigger it with an http client. In Azure ML workspace, everything works ok when executed. However, the experiment times out and fails when I trigger the experiment using an http client. Setting a timeout value for the http client does not seem to work.
Is there any way we can set this timeout value so that the experiment does not fail?

Make sure you're setting the client timeout value correctly. If the server powering the web service times out, then it will send back a response with the HTTP status code 504 BackendScoreTimeout (or possibly 409 GatewayTimeout). However, if you simply never receive a response, then your client isn't waiting long enough.
You can find out a good amount of time by running your experiment in ML Studio. Go to the experiment properties to find out how long it ran for, and then aim for about twice that amount of time as a timeout value.

I've had similar problems with an Azure ML experiment published as a web service. Most of the times it was running ok, while sometimes it returned with a timeout error. The problem is that the experiment itself has a 90 seconds running time limit. So, most probably your experiment has a running time over this limit and returns with a timeout error. hth

Looks like it isn't possible to set this timeout based on a feature request that is still marked as "planned" as of 4/1/2018.
The recommendation from MSDN forums from 2017 is to use the Batch Execution Service, which starts the machine learning experiment and then asynchronously asks whether it's done.
Here's a code snippet from the Azure ML Web Services Management Sample Code (all comments are from their sample code):
using (HttpClient client = new HttpClient())
{
var request = new BatchExecutionRequest()
{
Outputs = new Dictionary<string, AzureBlobDataReference> () {
{
"output",
new AzureBlobDataReference()
{
ConnectionString = storageConnectionString,
RelativeLocation = string.Format("{0}/outputresults.file_extension", StorageContainerName) /*Replace this with the location you would like to use for your output file, and valid file extension (usually .csv for scoring results, or .ilearner for trained models)*/
}
},
},
GlobalParameters = new Dictionary<string, string>() {
}
};
client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", apiKey);
// WARNING: The 'await' statement below can result in a deadlock
// if you are calling this code from the UI thread of an ASP.Net application.
// One way to address this would be to call ConfigureAwait(false)
// so that the execution does not attempt to resume on the original context.
// For instance, replace code such as:
// result = await DoSomeTask()
// with the following:
// result = await DoSomeTask().ConfigureAwait(false)
Console.WriteLine("Submitting the job...");
// submit the job
var response = await client.PostAsJsonAsync(BaseUrl + "?api-version=2.0", request);
if (!response.IsSuccessStatusCode)
{
await WriteFailedResponse(response);
return;
}
string jobId = await response.Content.ReadAsAsync<string>();
Console.WriteLine(string.Format("Job ID: {0}", jobId));
// start the job
Console.WriteLine("Starting the job...");
response = await client.PostAsync(BaseUrl + "/" + jobId + "/start?api-version=2.0", null);
if (!response.IsSuccessStatusCode)
{
await WriteFailedResponse(response);
return;
}
string jobLocation = BaseUrl + "/" + jobId + "?api-version=2.0";
Stopwatch watch = Stopwatch.StartNew();
bool done = false;
while (!done)
{
Console.WriteLine("Checking the job status...");
response = await client.GetAsync(jobLocation);
if (!response.IsSuccessStatusCode)
{
await WriteFailedResponse(response);
return;
}
BatchScoreStatus status = await response.Content.ReadAsAsync<BatchScoreStatus>();
if (watch.ElapsedMilliseconds > TimeOutInMilliseconds)
{
done = true;
Console.WriteLine(string.Format("Timed out. Deleting job {0} ...", jobId));
await client.DeleteAsync(jobLocation);
}
switch (status.StatusCode) {
case BatchScoreStatusCode.NotStarted:
Console.WriteLine(string.Format("Job {0} not yet started...", jobId));
break;
case BatchScoreStatusCode.Running:
Console.WriteLine(string.Format("Job {0} running...", jobId));
break;
case BatchScoreStatusCode.Failed:
Console.WriteLine(string.Format("Job {0} failed!", jobId));
Console.WriteLine(string.Format("Error details: {0}", status.Details));
done = true;
break;
case BatchScoreStatusCode.Cancelled:
Console.WriteLine(string.Format("Job {0} cancelled!", jobId));
done = true;
break;
case BatchScoreStatusCode.Finished:
done = true;
Console.WriteLine(string.Format("Job {0} finished!", jobId));
ProcessResults(status);
break;
}
if (!done) {
Thread.Sleep(1000); // Wait one second
}
}
}

.NET HttpClient.PostAsync() slow after 3 requests

I am using the .NET 4.5 HttpClient class to make a POST request to a server a number of times. The first 3 calls run quickly, but the fourth time a call to await client.PostAsync(...) is made, it hangs for several seconds before returning the expected response.
using (HttpClient client = new HttpClient())
{
// Prepare query
StringBuilder queryBuilder = new StringBuilder();
queryBuilder.Append("?arg=value");
// Send query
using (var result = await client.PostAsync(BaseUrl + queryBuilder.ToString(),
new StreamContent(streamData)))
{
Stream stream = await result.Content.ReadAsStreamAsync();
return new MyResult(stream);
}
}
The server code is shown below:
HttpListener listener;
void Run()
{
listener.Start();
ThreadPool.QueueUserWorkItem((o) =>
{
while (listener.IsListening)
{
ThreadPool.QueueUserWorkItem((c) =>
{
var context = c as HttpListenerContext;
try
{
// Handle request
}
finally
{
// Always close the stream
context.Response.OutputStream.Close();
}
}, listener.GetContext());
}
});
}
Inserting a debug statement at // Handle request shows that the server code doesn't seem to receive the request as soon as it is sent.
I have already investigated whether it could be a problem with the client not closing the response, meaning that the number of connections the ServicePoint provider allows could be reached. However, I have tried increasing ServicePointManager.MaxServicePoints but this has no effect at all.
I also found this similar question:
.NET HttpClient hangs after several requests (unless Fiddler is active)
I don't believe this is the problem with my code - even changing my code to exactly what is given there didn't fix the problem.

The problem was that there were too many Task instances scheduled to run.
Changing some of the Task.Factory.StartNew calls in my program for tasks which ran for a long time to use the TaskCreationOptions.LongRunning option fixed this. It appears that the task scheduler was waiting for other tasks to finish before it scheduled the request to the server.

Let code run for X seconds, after that stop

I'm working on Windows Phone 7.1 app and want for several lines of code run for 10 seconds, if succeeds in 10 seconds, continue, if no success, stop the code and display message.
The thing is, my code is not a loop - phone tries to fetch data from a server (if internet connection is slow, might take too long).
if (DeviceNetworkInformation.IsNetworkAvailable)
{
// start timer here for 10s
WebClient webClient = new WebClient();
webClient.DownloadStringCompleted += loginHandler;
webClient.DownloadStringAsync(new Uri(string.Format(url + "?loginas=" + login + "&pass=" + pwd)));
// if 10s passed, stop code above and display MessageBox
}

You can use something like the following:
HttpClient client = new HttpClient();
var cts = new CancellationTokenSource();
cts.CancelAfter(10000);
try
{
var response = await client.GetAsync(new Uri(string.Format(url +
"?loginas=" + login + "&pass=" + pwd)), cts.Token);
var result = await response.Content.ReadAsStringAsync();
// work with result
}
catch(TaskCanceledException)
{
// error/timeout handling
}
You need the follwoing NuGet packages:
HttpClient
async/await

Make that piece of code a method, make that method run separately.
Launch a Timer, when 10 seconds elapsed, check the status of the first thread.
If it has fetched all that he was supposed to, make use of that, otherwise kill the thread, clean whatever you have to clean and return that message of error.