We Keep Coding

Yet Another HttpClient / Async Deadlock

I've been banging my head on this for hours, and I've thrown my hands up in the air on this one. As far as I can tell, I've encountered a deadlock surrounding HttpClient and Async.
The goal is for a series of unrelated Posts to all get fired off in rapid succession, wait for them all to get done, and then construct a document from the resultset. The program has a WPF UI, and this is triggered by button:
private async void Generate_Suite_BTN_Click(object sender, RoutedEventArgs e)
{
var suiteBuilder = new SuiteBuilder();
await Task.Run(() => suiteBuilder.worker_Run());
}
That triggers the worker_Run(), which has some switch logic, and eventually gets down to hitting SendFiles() which has a Parrallel.Foreach, since the sending of the files doesn't need to be sequential, and are unrelated to eachother:
private bool SendFiles()
{
var result = Parallel.ForEach(_InfoCollection, SendFile);
return result.IsCompleted;
}
Each of those (in the parallel), is awaited in SendFile(), which also has some switch logic, based on what we're sending, but ultimately boils down to:
var result = await Client.SendMessage ( vars );
results.Add(result.MessageId, result.StatusCode, result.HttpReason);
Here's the HttpClient portion in SendMessage():
public async Task<Result> SendMessage(vars)
{
var soapResponse = new XmlDocument();
try
{
Client.DefaultRequestHeaders.Add("SOAPAction", soapAction);
Client.Timeout = TimeSpan.FromSeconds(Timeout);
var content = new StringContent(soapRequest, Encoding.UTF8, contentType);
var post = await Client.PostAsync(url, content).ConfigureAwait(false);
var response = post.Content;
result.StatusCode = post.StatusCode;
result.HttpReason = post.ReasonPhrase;
var sResponse = await response.ReadAsStringAsync().ConfigureAwait(false);
soapResponse.Load(sResponse);
}
catch (Exception ex)
{
//Catch logic
}
}
I can see the request and response passing back and forth with Fiddler, but I'm having trouble line-by-line debugging, since as soon as I hit the PostAsync, VS flips out and continues all the way to the end of the program, skipping all breakpoints. Meanwhile, the request times out with a TaskCanceledException, long after the code that should come afterwards already has completed.
I've looked a dozens of question & answers here on SO, and elsewhere, but they're just not quite helping find the problem. Most seem to center around the liberal sprinkling of ".ConfigureAwait(false)" on Async calls, but it doesn't really seem to have any effect.

So, with the help of #JSteward in the comments, he pointed out that async and Parrallel.ForEach aren't well suited to working together, since return types of void when dealing with async should be avoided.
He recommended that I use Async only, from the top (Button Click) to the bottom (Message Send), and that worked out. Credit to him for the guidance.
This link was helpful in explaining why this was the case: Async/Await - Best Practices in Asynchronous Programming
SendFiles ended up looking something like this:
private async Task<bool> SendFiles()
{
var result = _InfoCollection.Select(SendFile);
await Task.WhenAll(result).ConfigureAwait(false);
return true;
}
All the other methods went async, with appropriate awaits, and return types of Task or Task<T>.

Difference between DownloadStringTaskAsync & DownloadStringAsync

I am working on an asp.net mvc5 web application, and i am not sure what are the differences between using DownloadStringTaskAsync() & usingDownloadStringAsync() . for example if i have the following webclient :-
using (WebClient wc = new WebClient())
{
string url = currentURL + "home/scanserver?tokenfromtms=" + "12345" + "&FQDN=allscan" ;
var json = await wc.DownloadStringTaskAsync(url);
TempData["messagePartial"] = string.Format("Scan has been completed. Scan reported generated");
}
will there be any differences if i chnage DownloadStringTaskAsync(url); to DownloadStringAsync(url); ??

WebClient.DownloadStringAsync is using the older event-based asynchronous pattern (EAP).
WebClient.DownloadStringTaskAsync is using the newer task-based asynchronous pattern (TAP).
Since your code is already using async/await, I recommend you stick with the TAP method. TAP code is more maintainable than EAP code.
You could also go another step further and consider using HttpClient instead of WebClient. The odd naming in WebClient is due to it supporting both synchronous and EAP, and then later updated to include TAP. In contrast, HttpClient is a newer type that was based on TAP right from the beginning, so its API is cleaner.

The DownloadStringTaskAsync and DownloadStringAsync documentation do a pretty good job of highlighting both similarities and differences.
They both are non-blocking, asynchronous methods. However, DownloadStringAsync has a return signature of void and requires you to listen to the DownloadStringCompleted event to obtain your results from Result, whereas the DownloadStringTaskAsync method returns a Task<string>.
The latter is useful if you have parallel asynchronous operations that you need to await before continuing or if you want to call ContinueWith on the operation once it's completed. In addition, with the latter you'll also need to retrieve the result from the task once the task is in a completed state which can be unwrapped with await.
Finally, DownloadStringAsync requires a URI whereas DownloadStringTaskAsync will accept a string.
For ease of use, DownloadStringTaskAsync will probably work just fine, provided that you've placed it in an async method as follows:
void Main()
{
using (WebClient wc = new WebClient())
{
var json = GetGoogleFromTask(wc);
json.Dump();
}
}
public async Task<string> GetGoogleFromTask(WebClient wc)
{
string url = "http://www.google.com" ;
var json = await wc.DownloadStringTaskAsync(url);
return json;
}
Alternatively, you can also return just the Task so that you can continue other operations without needing an async method that awaits the return:
void Main()
{
using (WebClient wc = new WebClient())
{
var json = GetGoogleFromTask(wc);
json.Dump();
}
}
public Task<string> GetGoogleFromTask(WebClient wc)
{
string url = "http://www.google.com" ;
var json = wc.DownloadStringTaskAsync(url);
return json;
}

Async Loop There is no longer an HttpContext available

I have a requirement, is to process X number of files, usually we can receive around 100 files each day, is a zip file so I have to open it, create a stream then send it to a WebApi service which is a workflow, this workflow calls two more WebApi Steps.
I implemented a console application that loops through the files then calls a wrapper which makes a REST call using HttpWebRequest.GetResponse().
I stressed tested the solution and created 11K files, in a synchronous version it takes to process all the files around 17 minutes, but I would like to create an async version of it and be able to use await HttpWebRequest.GetResponseAsync().
Here is the Async version:
private async Task<KeyValuePair<HttpStatusCode, string>> REST_CallAsync(
string httpMethod,
string url,
string contentType,
object bodyMessage = null,
Dictionary<string, object> headerParameters = null,
object[] queryStringParamaters = null,
string requestData = "")
{
try
{
HttpWebRequest req = (HttpWebRequest)HttpWebRequest.Create("some url");
req.Method = "POST";
req.ContentType = contentType;
//Adding zip stream to body
var reqBodyBytes = ReadFully((Stream)bodyMessage);
req.ContentLength = reqBodyBytes.Length;
Stream reqStream = req.GetRequestStream();
reqStream.Write(reqBodyBytes, 0, reqBodyBytes.Length);
reqStream.Close();
//Async call
var resp = await req.GetResponseAsync();
var httpResponse = (HttpWebResponse)resp as HttpWebResponse;
var responseData = new StreamReader(resp.GetResponseStream()).ReadToEnd();
return new KeyValuePair<HttpStatusCode,string>(httpResponse.StatusCode, responseData);
}
catch (WebException webEx)
{
//something
}
catch (Exception ex)
{
//something
}
In my console Application I have a loop to open and call the async (CallServiceAsync under the covers calls the method above)
foreach (var zipFile in Directory.EnumerateFiles(directory))
{
using (var zipStream = System.IO.File.OpenRead(zipFile))
{
await _restFulService.CallServiceAsync<WorkflowResponse>(
zipStream,
headerParameters,
null,
true);
}
processId++;
}
}
What end up happening was that only 2K of 11K got processed and didn't throw any exception so I was clueless so I changed the version I am calling the async to:
foreach (var zipFile in Directory.EnumerateFiles(directory))
{
using (var zipStream = System.IO.File.OpenRead(zipFile))
{
tasks.Add(_restFulService.CallServiceAsync<WorkflowResponse>(
zipStream,
headerParameters,
null,
true));
}
}
}
And have another loop to await for the tasks:
foreach (var task in await System.Threading.Tasks.Task.WhenAll(tasks))
{
if (task.Value != null)
{
Console.WriteLine("Ending Process");
}
}
And now I am facing a different error, when I process three files, the third one receives:
The client is disconnected because the underlying request has been completed. There is no longer an HttpContext available.
My question is, what i am doing wrong here? I use SimpleInjector as IoC would it be this the problem?
Also when you do WhenAll is waiting for each thread to run? Is not making it synchronous so it waits for a thread to finish in order to execute the next one? I am new to this async world so any help would be really much appreciated.

Well for those that added -1 to my question and instead of providing some type of solution just suggested something meaningless, here it is the answer and the reason why specifying as much detail as possible is useful.
First problem, since I'm using IIS Express if I'm not running my solution (F5) then the web applications are not available, that happened to me sometimes not always.
The second problem and the one giving me a huge headache is that not all the files got processed, I should've known the reason of this issue before, is the usage of async - await in a console application. I forced my console app to work with async by doing:
static void Main(string[] args)
{
System.Threading.Tasks.Task.Run(() => MainAsync(args)).Wait();
}
static async void MainAsync(string[] args)
{
//rest of code
Then if you note in my foreach I had await keyword and what was happening is that by concept await sends back the control flow to the caller, in this case the OS is the one calling the Console App (that is why doesn't make too much sense to use async - await in a console app, I did it because I mistakenly used await by calling an async method).
So the result was that my process only processed some X number of files, so what I end up doing is the following:
Add a list of tasks, the same way I did above:
tasks.Add(_restFulService.CallServiceAsync<WorkflowResponse>(....
And the way to run the threads is (in my console app):
ExecuteAsync(tasks);
Finally my method:
static void ExecuteAsync(List<System.Threading.Tasks.Task<KeyValuePair<HttpStatusCode, WorkflowResponse>>> tasks)
{
System.Threading.Tasks.Task.WhenAll(tasks).Wait();
}
UPDATE: Based on Scott's feedback, I changed the way I execute my threads.
And now I'm able to process all my files, I tested it and to process 1000 files in my synchronous process took around 160+ seconds to run all the process (I have a workflow of three steps in order to process the file) and when I put my async process in place it took 80+ seconds so almost half of the time. In my production server with IIS I believe the execution time will be less.
Hope this helps to anyone facing this type of issue.

WP8 Async/Await begingetresponse not waiting, gets run last

I may be misunderstanding the flow of control, because by all accounts this seems like it should work. This is a Windows phone 8 app. I am attempting to make a web request, and accordingly display the data returned. I am trying to get the data (here, called 'Key') in the following method:
public Task<String> getSingleStockQuote(String URI)
{
return Task.Run(() =>
{
String key = null;
HttpWebRequest request = HttpWebRequest.Create(URI) as HttpWebRequest;
HttpWebResponse response;
try
{
request.BeginGetResponse((asyncres) =>
{
HttpWebRequest responseRequest = (HttpWebRequest)asyncres.AsyncState;
response = (HttpWebResponse)responseRequest.EndGetResponse(asyncres);
key = returnStringFromStream(response.GetResponseStream());
System.Diagnostics.Debug.WriteLine(key);
}, request);
}
catch (Exception e)
{
System.Diagnostics.Debug.WriteLine("WebAccessRT getSingleStockQuote threw exception");
key = String.Empty;
}
return key;
});
}
...And I am calling this method like so:
WebAccessRT rt = new WebAccessRT();
await rt.getSingleStockQuote(stockTagURI);
System.Diagnostics.Debug.WriteLine("Past load data");
The WriteLine() in BeginGetResponse is for testing purposes; it prints after "Past Load Data". I want BeginGetResponse to run and complete operation (thus setting the Key), before the task returns. The data prints out right in the console, but not in the desired order - so Key is set and has a value, but its the very last part that gets run. Can someone point me in the right direction and/or see what's causing the problem above? Thinking this through, is the await operator SIMPLY waiting for the Task to return, which is returning after spinning off its async call?

BeginGetResponse starts an asynchronous process (hence the callback) so you cannot guarantee the order it is completed. Remember that the code within BeginGetResponse is actually a separate method (see closures) that is executed separately from getSingleStockQuote. You would need to use await GetResponseAsync or (imo, even better - you could greatly simplify your code) use HttpClient.

WP7 synchronous web request

I’m developing a Windows Phone 7.1 application, and trying to implement tombstoning.
Due to the legal reasons I can’t save my view model. I’m only saving encrypted session ID, which can be used to load a view model data from the remote server.
On resume, I need to verify the session ID, if it’s expired – I take user to the login page of my app, if it’s still OK, I reload view model data from the server.
The problem is the HttpWebRequest lacks blocking API. Moreover, while inside page.OnNavigatedTo method after de-tombstoning, the method described here blocks forever.
I’ve worked around the problem by presenting my own splash screen.
However, I’d rather like to complete those RPC calls while the system-provided “Resuming…” splash screen is visible, i.e. before I return from page.OnNavigatedTo method.
Any ideas how can I complete HTTP requests synchronously while inside page.OnNavigatedTo after de-tombstoning?

Let me start out by saying that Microsoft really tries to push you to do async calls for good reasons, which is why I wanted to emphasize it.
Now if you really want to do it synchronous, I have an idea which I haven't been able to test myself. When using the HttpWebRequest class, there are two important functions, which you've probably used as well: BeginGetResponse and EndGetResponse.
These two functions work closely together. BeginGetResponse starts a asynchronous webrequest, where when the request is finished the EndGetResponse gives you to ouput when it's done. This is the way MS tries to let you do it. The trick to doing this stuff synchronously is that the beginGetResponse returns a IAsyncResult. This IAsyncResult interface contains a WaitHandler, which can be used to synchronously wait until the request is done. After which you can just continue with the endGetRequest and go on with your bussiness. The same thing goes for the BeginGetRequestStream and EndGetRequestStream.
But as I said before, I haven't tested this solution and it's purely theoretical. Let me know if it worked or not.
Good luck!

Update: another option is to use Reactive Extensions.
If you're on VS2010 you can install the AsyncCTP and when you do an extension method gets added that allows you to await the response.
static async Task<Stream> AsynchronousDownload(string url)
{
WebRequest request = WebRequest.Create(url);
WebResponse response = await request.GetResponseAsync();
return (response.GetResponseStream());
}
then:
UPDATED:
protected override async void OnNavigatedTo(NavigationEventArgs e)
{
base.OnNavigatedTo(e);
var myResponse = await AsynchronousDownload("http://stackoverflow.com");
}
or
If you're using VS2012 you can install the Microsoft.Bcl.Async lib and do the same thing as if you were using the AsyncCTP, await the response.
or
You could implement something similar to Coroutines in Caliburn Micro. For this you implement the IResult interface.
public interface IResult
{
void Execute(ActionExecutionContext context);
event EventHandler<ResultCompletionEventArgs> Completed;
}
A possible implementation:
public class HttpWebRequestResult : IResult
{
public HttpWebRequest HttpWebRequest { get; set; }
public string Result { get; set; }
public HttpWebRequestResult(string url)
{
HttpWebRequest = (HttpWebRequest) HttpWebRequest.Create(url);
}
public void Execute (ActionExecutionContext context)
{
HttpWebRequest.BeginGetResponse (Callback, HttpWebRequest);
}
public void Callback (IAsyncResult asyncResult)
{
var httpWebRequest = (HttpWebRequest)asyncResult.AsyncState;
var httpWebResponse = (HttpWebResponse) httpWebRequest.EndGetResponse(asyncResult);
using (var reader = new StreamReader(httpWebResponse.GetResponseStream()))
Result = reader.ReadToEnd();
Completed (this, new ResultCompletionEventArgs ());
}
public event EventHandler<ResultCompletionEventArgs> Completed = delegate { };
}
Then to call it:
var httpWebRequestResult = new HttpWebRequestResult("http://www.google.com");
yield return httpWebRequestResult;
var result = httpWebRequestResult.Result;
This might be an example of grabbing the Coroutines implementation from CM and using it separately.