I have an application that sends a lot of httpwebrequests and reads data. So I decided to run each request on a different task. I ended up with having about 60 tasks to run. the problem is that the these requests does not get executed when I use task.run. In fact some of the requests gets sent to the server and responses back with data and a lot of them are just ignored. I was wondering if this is because of the CPU limitations and if so how would it be possible to handle such requests.
Example of the the method that requests data:
private static bool getResponseData(string url, string pageNumber)
{
url = url + "page=" + pageNumber;
Uri myUri = new Uri(url, UriKind.Absolute);
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(myUri);
request.AutomaticDecompression = DecompressionMethods.GZip;
try
{
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
if (response.StatusCode == HttpStatusCode.OK)
{
Stream receiveStream = response.GetResponseStream();
StreamReader readStream = null;
if (String.IsNullOrWhiteSpace(response.CharacterSet))
readStream = new StreamReader(receiveStream);
else
readStream = new StreamReader(receiveStream, Encoding.GetEncoding(response.CharacterSet));
string data = readStream.ReadToEnd();
//dealing with data by storing it in the data base
return true;
}
return false;
}
}
}
A method to loop throughout url pages
private void startrequest(string url)
{
bool data2 = getResponseData(url, "0");
int counter2 = 0;
while (data2 && counter2 < 4)
{
counter2++;
data2 = getResponseData(url, counter2.ToString());
}
}
Example of the use of the task.run in my application is as follows:
Task task1 = Task.Run(() => startrequest("www.x1.com"));
Task task2 = Task.Run(() => startrequest("www.x2.com"));
Task task3 = Task.Run(() => startrequest("www.x3.com"));
Task task4 = Task.Run(() => startrequest("www.x4.com"));
Task.WaitAll(task1, task2, task3, task4);
if I run the method startrequest manually without the use of the Task.Run I always end up by getting the correct data. I am running my PC on I5.
Related
Using form contenttype, I am getting stuck at this line:
var webresponse = (HttpWebResponse)webrequest.GetResponse();
For the url, you can use any value. Here is the full code:
public bool test()
{
string url = "https://www.google.com";
// Create a new HTTP request object, set the method to POST and write the POST data to it
var webrequest = (HttpWebRequest)WebRequest.CreateHttp(url);
webrequest.Method = "POST";
webrequest.ContentType = "application/x-www-form-urlencoded";
using (Stream postStream = webrequest.GetRequestStream())
{
//
}
// Make the request, get a response and pull the data out of the response stream
var webresponse = (HttpWebResponse)webrequest.GetResponse();
Stream responseStream = webresponse.GetResponseStream();
var reader = new StreamReader(responseStream);
string result = reader.ReadToEnd();
// Normal Completion
return true;
}
Does anyone know why it would get stuck on this line and not come back (when you are doing debugging):
var webresponse = (HttpWebResponse)webrequest.GetResponse();
Thank You
I found the problem! I was calling the method as an async task:
private async Task starttest()
{
clsTester objTester = new clsTester();
objTester.test();
}
...
var result = Task.Run(async () => await starttest());
I removed all that code and changed it to a worker thread:
private Thread workerThread = null;
private void starttest()
{
clsTester objTester = new clsTester();
objTester.test();
}
And then called it as follows:
...
// Initialise and start worker thread
this.workerThread = new Thread(new ThreadStart(this.starttest));
this.workerThread.Start();
Everything is working now.
I am running a Task, which copies from one stream to another. This works without problems, including progress reporting. But i cant cancel the task. If i fire the CancellationToken, the copy progress runs till its completion, then the task is cancelled, but this is of course to late. Here is my actual code
private async Task Download(Uri uriToWork, CancellationToken cts)
{
HttpClient httpClient = new HttpClient();
HttpRequestMessage requestAction = new HttpRequestMessage();
requestAction.Method = new HttpMethod("GET");
requestAction.RequestUri = uriToWork;
HttpResponseMessage httpResponseContent = await httpClient.SendRequestAsync(requestAction, HttpCompletionOption.ResponseHeadersRead);
using (Stream streamToRead = (await httpResponseContent.Content.ReadAsInputStreamAsync()).AsStreamForRead())
{
string fileToWrite = Path.GetTempFileName();
using (Stream streamToWrite = File.Open(fileToWrite, FileMode.Create))
{
await httpResponseContent.Content.WriteToStreamAsync(streamToWrite.AsOutputStream()).AsTask(cts, progressDownload);
await streamToWrite.FlushAsync();
//streamToWrite.Dispose();
}
await streamToRead.FlushAsync();
//streamToRead.Dispose();
}
httpClient.Dispose();
}
Can someone help me please, or can explain, why it does not work?
Is it this operation that continues until it completes ?
await httpResponseContent.Content.WriteToStreamAsync(streamToWrite.AsOutputStream()).AsTask(cts, progressDownload);
Or is it this one ?
await streamToWrite.FlushAsync();
I think the latter needs probably to have the CancellationToken as well:
await streamToWrite.FlushAsync(cts);
Unfortunately I cannot answer why this cancel does not occur. However, a solution that consists in writing the Stream in chunks may help.
Here is something very dirty that works:
private async Task Download(Uri uriToWork, CancellationToken cts) {
using(HttpClient httpClient = new HttpClient()) {
HttpRequestMessage requestAction = new HttpRequestMessage();
requestAction.Method = new HttpMethod("GET");
requestAction.RequestUri = uriToWork;
HttpResponseMessage httpResponseContent = await httpClient.SendRequestAsync(requestAction, HttpCompletionOption.ResponseHeadersRead);
string fileToWrite = Path.GetTempFileName();
using(Stream streamToWrite = File.Open(fileToWrite, FileMode.Create)) {
// Disposes streamToWrite to force any write operation to fail
cts.Register(() => streamToWrite.Dispose());
try {
await httpResponseContent.Content.WriteToStreamAsync(streamToWrite.AsOutputStream()).AsTask(cts, p);
}
catch(TaskCanceledException) {
return; // "gracefully" exit when the token is cancelled
}
await streamToWrite.FlushAsync();
}
}
}
I enclosed the httpClient in a using so a return disposes it properly.
I removed the streamToRead which was not used at all
Now here is the horror: I added a delegate that executes when the token is cancelled: it disposes streamToWrite while it is written to (ughhhh), which triggers an TaskCancelledException when WriteToStreamAsync cannot longer write in this disposed stream.
Please dont throw a puke bag at me yet, I am not experienced enough in this "Universal" Framework which looks very different as the usual one.
Here is a chunked stream solution that looks more acceptable. I shortened a bit the original code and added the IProgress as a parameter.
async Task Download(Uri uriToWork, CancellationToken cts, IProgress<int> progress) {
using(HttpClient httpClient = new HttpClient()) {
var chunkSize = 1024;
var buffer = new byte[chunkSize];
int count = 0;
string fileToWrite = Path.GetTempFileName();
using(var inputStream = await httpClient.GetInputStreamAsync(uriToWork)) {
using(var streamToRead = inputStream.AsStreamForRead()) {
using(Stream streamToWrite = File.OpenWrite(fileToWrite)) {
int size;
while((size = await streamToRead.ReadAsync(buffer, 0, chunkSize, cts).ConfigureAwait(false)) > 0) {
count += size;
await Dispatcher.RunAsync(CoreDispatcherPriority.Normal, () => progress.Report(count));
// progress.Report(count);
await streamToWrite.WriteAsync(buffer, 0, size, cts).ConfigureAwait(false);
}
}
}
}
}
}
The blocking operation is most probably not WriteToStreamAsync() but FlushAsync(), so #Larry's assumption should be right, the FlushAsync method needs the cancellation token as well.
How can I use HttpWebRequest (.NET, C#) asynchronously?
Use HttpWebRequest.BeginGetResponse()
HttpWebRequest webRequest;
void StartWebRequest()
{
webRequest.BeginGetResponse(new AsyncCallback(FinishWebRequest), null);
}
void FinishWebRequest(IAsyncResult result)
{
webRequest.EndGetResponse(result);
}
The callback function is called when the asynchronous operation is complete. You need to at least call EndGetResponse() from this function.
By far the easiest way is by using TaskFactory.FromAsync from the TPL. It's literally a couple of lines of code when used in conjunction with the new async/await keywords:
var request = WebRequest.Create("http://www.stackoverflow.com");
var response = (HttpWebResponse) await Task.Factory
.FromAsync<WebResponse>(request.BeginGetResponse,
request.EndGetResponse,
null);
Debug.Assert(response.StatusCode == HttpStatusCode.OK);
If you can't use the C#5 compiler then the above can be accomplished using the Task.ContinueWith method:
Task.Factory.FromAsync<WebResponse>(request.BeginGetResponse,
request.EndGetResponse,
null)
.ContinueWith(task =>
{
var response = (HttpWebResponse) task.Result;
Debug.Assert(response.StatusCode == HttpStatusCode.OK);
});
Considering the answer:
HttpWebRequest webRequest;
void StartWebRequest()
{
webRequest.BeginGetResponse(new AsyncCallback(FinishWebRequest), null);
}
void FinishWebRequest(IAsyncResult result)
{
webRequest.EndGetResponse(result);
}
You could send the request pointer or any other object like this:
void StartWebRequest()
{
HttpWebRequest webRequest = ...;
webRequest.BeginGetResponse(new AsyncCallback(FinishWebRequest), webRequest);
}
void FinishWebRequest(IAsyncResult result)
{
HttpWebResponse response = (result.AsyncState as HttpWebRequest).EndGetResponse(result) as HttpWebResponse;
}
Greetings
Everyone so far has been wrong, because BeginGetResponse() does some work on the current thread. From the documentation:
The BeginGetResponse method requires some synchronous setup tasks to
complete (DNS resolution, proxy detection, and TCP socket connection,
for example) before this method becomes asynchronous. As a result,
this method should never be called on a user interface (UI) thread
because it might take considerable time (up to several minutes
depending on network settings) to complete the initial synchronous
setup tasks before an exception for an error is thrown or the method
succeeds.
So to do this right:
void DoWithResponse(HttpWebRequest request, Action<HttpWebResponse> responseAction)
{
Action wrapperAction = () =>
{
request.BeginGetResponse(new AsyncCallback((iar) =>
{
var response = (HttpWebResponse)((HttpWebRequest)iar.AsyncState).EndGetResponse(iar);
responseAction(response);
}), request);
};
wrapperAction.BeginInvoke(new AsyncCallback((iar) =>
{
var action = (Action)iar.AsyncState;
action.EndInvoke(iar);
}), wrapperAction);
}
You can then do what you need to with the response. For example:
HttpWebRequest request;
// init your request...then:
DoWithResponse(request, (response) => {
var body = new StreamReader(response.GetResponseStream()).ReadToEnd();
Console.Write(body);
});
public static async Task<byte[]> GetBytesAsync(string url) {
var request = (HttpWebRequest)WebRequest.Create(url);
using (var response = await request.GetResponseAsync())
using (var content = new MemoryStream())
using (var responseStream = response.GetResponseStream()) {
await responseStream.CopyToAsync(content);
return content.ToArray();
}
}
public static async Task<string> GetStringAsync(string url) {
var bytes = await GetBytesAsync(url);
return Encoding.UTF8.GetString(bytes, 0, bytes.Length);
}
I ended up using BackgroundWorker, it is definitely asynchronous unlike some of the above solutions, it handles returning to the GUI thread for you, and it is very easy to understand.
It is also very easy to handle exceptions, as they end up in the RunWorkerCompleted method, but make sure you read this: Unhandled exceptions in BackgroundWorker
I used WebClient but obviously you could use HttpWebRequest.GetResponse if you wanted.
var worker = new BackgroundWorker();
worker.DoWork += (sender, args) => {
args.Result = new WebClient().DownloadString(settings.test_url);
};
worker.RunWorkerCompleted += (sender, e) => {
if (e.Error != null) {
connectivityLabel.Text = "Error: " + e.Error.Message;
} else {
connectivityLabel.Text = "Connectivity OK";
Log.d("result:" + e.Result);
}
};
connectivityLabel.Text = "Testing Connectivity";
worker.RunWorkerAsync();
.NET has changed since many of these answers were posted, and I'd like to provide a more up-to-date answer. Use an async method to start a Task that will run on a background thread:
private async Task<String> MakeRequestAsync(String url)
{
String responseText = await Task.Run(() =>
{
try
{
HttpWebRequest request = WebRequest.Create(url) as HttpWebRequest;
WebResponse response = request.GetResponse();
Stream responseStream = response.GetResponseStream();
return new StreamReader(responseStream).ReadToEnd();
}
catch (Exception e)
{
Console.WriteLine("Error: " + e.Message);
}
return null;
});
return responseText;
}
To use the async method:
String response = await MakeRequestAsync("http://example.com/");
Update:
This solution does not work for UWP apps which use WebRequest.GetResponseAsync() instead of WebRequest.GetResponse(), and it does not call the Dispose() methods where appropriate. #dragansr has a good alternative solution that addresses these issues.
public void GetResponseAsync (HttpWebRequest request, Action<HttpWebResponse> gotResponse)
{
if (request != null) {
request.BeginGetRequestStream ((r) => {
try { // there's a try/catch here because execution path is different from invokation one, exception here may cause a crash
HttpWebResponse response = request.EndGetResponse (r);
if (gotResponse != null)
gotResponse (response);
} catch (Exception x) {
Console.WriteLine ("Unable to get response for '" + request.RequestUri + "' Err: " + x);
}
}, null);
}
}
Follow up to the #Isak 's answer, which is very good. Nonetheless it's biggest flaw is that it will only call the responseAction if the response has status 200-299. The best way to fix this is:
private void DoWithResponseAsync(HttpWebRequest request, Action<HttpWebResponse> responseAction)
{
Action wrapperAction = () =>
{
request.BeginGetResponse(new AsyncCallback((iar) =>
{
HttpWebResponse response;
try
{
response = (HttpWebResponse)((HttpWebRequest)iar.AsyncState).EndGetResponse(iar);
}
catch (WebException ex)
{
// It needs to be done like this in order to read responses with error status:
response = ex.Response as HttpWebResponse;
}
responseAction(response);
}), request);
};
wrapperAction.BeginInvoke(new AsyncCallback((iar) =>
{
var action = (Action)iar.AsyncState;
action.EndInvoke(iar);
}), wrapperAction);
}
And then as #Isak follows:
HttpWebRequest request;
// init your request...then:
DoWithResponse(request, (response) => {
var body = new StreamReader(response.GetResponseStream()).ReadToEnd();
Console.Write(body);
});
I've been using this for async UWR, hopefully it helps someone
string uri = "http://some.place.online";
using (UnityWebRequest uwr = UnityWebRequest.Get(uri))
{
var asyncOp = uwr.SendWebRequest();
while (asyncOp.isDone == false) await Task.Delay(1000 / 30); // 30 hertz
if(uwr.result == UnityWebRequest.Result.Success) return uwr.downloadHandler.text;
Debug.LogError(uwr.error);
}
Why won't this HTTPHandler work? Both getPageBody and getPageContent await forever and never get back to me. Nothing else happens after the await (used a break point).
Any help would be strongly appreciated!
PS: Visiting the page in the browser does work - so the problem must be on the C# end.
public class HTTPHandler
{
public static async Task<List<String>> getPageBody(String page)
{
WebRequest request = WebRequest.Create(
"http://www.mywebsite.com/dev/api/" + page);
request.Credentials = CredentialCache.DefaultCredentials;
WebResponse response = await request.GetResponseAsync();
Stream dataStream = response.GetResponseStream();
StreamReader reader = new StreamReader(dataStream);
string responseFromServer = reader.ReadToEnd();
//reader.Close();
//response.Close();
return responseFromServer.Split(';').ToList();
}
public static async Task<List<String>> getPageContents(String page)
{
HttpClient client = new HttpClient();
Task<HttpResponseMessage> resp;
await (resp = client.GetAsync("http://www.mywebsite.com/dev/api/" + page)).ContinueWith(
(getTask) =>
{
getTask.Result.EnsureSuccessStatusCode();
});
//HttpResponseMessage resp = await client.GetAsync("http://mywebsite.com/dev/api/" + page);
Task<String> responseBodyAsText = resp.Result.Content.ReadAsStringAsync();
responseBodyAsText.Wait();
return responseBodyAsText.Result.Split(';').ToList();
}
}
I suspect that further up your call stack, you're using Task.Wait or Task.Result.
If you use Wait or Result on async code, you run the risk of deadlock. I explain this in more detail on my blog and in a recent MSDN article.
I have this piece of code to make an asynchronous HTTP request:
public static void MakeRequest(Uri uri, Action<RequestCallbackState> responseCallback)
{
WebRequest request = WebRequest.Create(uri);
request.Proxy = null;
Task.Factory.FromAsync<WebResponse>(request.BeginGetResponse, request.EndGetResponse, null).ContinueWith(task =>
{
WebResponse response = task.Result;
Stream responseStream = response.GetResponseStream();
responseCallback(new RequestCallbackState(response.GetResponseStream()));
responseStream.Close();
response.Close();
});
}
It works, but I need to set a request timeout. I tried to use request.Timeout but don't seem to do anything. Is there a way to set up a task timeout in this code?
Edited to add a new timeout callback.
New code:
public static void MakeRequest(Uri uri, Action<RequestCallbackState> responseCallback)
{
WebRequest request = WebRequest.Create(uri);
request.Proxy = null;
IAsyncResult t = Task.Factory.FromAsync<WebResponse>(request.BeginGetResponse, request.EndGetResponse, null).ContinueWith(task =>
{
WebResponse response = task.Result;
Stream responseStream = response.GetResponseStream();
responseCallback(new RequestCallbackState(response.GetResponseStream()));
responseStream.Close();
response.Close();
});
ThreadPool.RegisterWaitForSingleObject(t.AsyncWaitHandle, new WaitOrTimerCallback(TimeoutCallback), request, 1000, true);
}
private static void TimeoutCallback(object state, bool timedOut)
{
if (timedOut)
{
Console.WriteLine("Timeout");
WebRequest request = (WebRequest)state;
if (state != null)
{
request.Abort();
}
}
}
Testing with:
HttpSocket.MakeRequest(new Uri("http://www.google.comhklhlñ"), callbackState =>
{
if (callbackState.Exception != null)
throw callbackState.Exception;
Console.WriteLine(GetResponseText(callbackState.ResponseStream));
});
Thread.Sleep(10000);
From the Timeout documentation:
The Timeout property has no effect on asynchronous requests made with the BeginGetResponse or BeginGetRequestStream method.
This is because the framework forces you to handle the timeout yourself. You should be able to use the example code here, except pass the Task returned from the FromAsync call to the ThreadPool.RegisterWaitForSingleObject method.
Edit:
You need to put the registration on the original task, not the continuation:
public static void MakeRequest(Uri uri, Action<Stream> responseCallback)
{
WebRequest request = WebRequest.Create(uri);
request.Proxy = null;
const int TimeoutPeriod = 1000;
Task<WebResponse> t = Task.Factory.FromAsync<WebResponse>(request.BeginGetResponse, request.EndGetResponse, null);
ThreadPool.RegisterWaitForSingleObject((t as IAsyncResult).AsyncWaitHandle, TimeoutCallback, request, TimeoutPeriod, true);
t.ContinueWith(task =>
{
WebResponse response = task.Result;
Stream responseStream = response.GetResponseStream();
responseCallback(response.GetResponseStream());
responseStream.Close();
response.Close();
});
}
This works for me (if I set the timeout duration very short, and hit this, I always timeout appropriately).