I have an application that sends a lot of httpwebrequests and reads data. So I decided to run each request on a different task. I ended up with having about 60 tasks to run. the problem is that the these requests does not get executed when I use task.run. In fact some of the requests gets sent to the server and responses back with data and a lot of them are just ignored. I was wondering if this is because of the CPU limitations and if so how would it be possible to handle such requests.
Example of the the method that requests data:
private static bool getResponseData(string url, string pageNumber)
{
url = url + "page=" + pageNumber;
Uri myUri = new Uri(url, UriKind.Absolute);
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(myUri);
request.AutomaticDecompression = DecompressionMethods.GZip;
try
{
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
if (response.StatusCode == HttpStatusCode.OK)
{
Stream receiveStream = response.GetResponseStream();
StreamReader readStream = null;
if (String.IsNullOrWhiteSpace(response.CharacterSet))
readStream = new StreamReader(receiveStream);
else
readStream = new StreamReader(receiveStream, Encoding.GetEncoding(response.CharacterSet));
string data = readStream.ReadToEnd();
//dealing with data by storing it in the data base
return true;
}
return false;
}
}
}
A method to loop throughout url pages
private void startrequest(string url)
{
bool data2 = getResponseData(url, "0");
int counter2 = 0;
while (data2 && counter2 < 4)
{
counter2++;
data2 = getResponseData(url, counter2.ToString());
}
}
Example of the use of the task.run in my application is as follows:
Task task1 = Task.Run(() => startrequest("www.x1.com"));
Task task2 = Task.Run(() => startrequest("www.x2.com"));
Task task3 = Task.Run(() => startrequest("www.x3.com"));
Task task4 = Task.Run(() => startrequest("www.x4.com"));
Task.WaitAll(task1, task2, task3, task4);
if I run the method startrequest manually without the use of the Task.Run I always end up by getting the correct data. I am running my PC on I5.
I have a progressBar on my form and a button.
When user clicks this button, the progressBar should be styled as "marquee" and the program begins to check is an URL is valid or not.. OK.
But when I click the button, the UI freezes until the HttpStatusCode returns true or false...
Here is the check code:
private bool RemoteFileExists(string url)
{
try
{
HttpWebRequest request = WebRequest.Create(url) as HttpWebRequest;
request.Method = "HEAD";
HttpWebResponse response = request.GetResponse() as HttpWebResponse;
return (response.StatusCode == HttpStatusCode.OK);
}
catch
{
return false;
}
}
And here is the button click code:
private async void button1_Click(object sender, EventArgs e)
{
this.progressBar1.Style = System.Windows.Forms.ProgressBarStyle.Marquee;
var result = RemoteFileExists("http://www.google.com/");
if (Completed)
{
//ok
}
else
{
//not ok
}
}
The UI freezes because you are executing the RemoteFileExists method on the UI thread and receiving a response from a HttpWebRequest takes some time.
To solve this you have to execute RemoteFileExists in a different thread than the UI thread.
As your button1_Click method is already declared async the easiest way would be to declare RemoteFileExists as async too.
Then you can use the HttpWebRequest.GetResponseAsync method to asynchronously receive the response object.
private async Task<bool> RemoteFileExists(string url)
{
try
{
HttpWebRequest request = WebRequest.CreateHttp(url);
request.Method = "HEAD";
using(var response = (HttpWebResponse) await request.GetResponseAsync())
{
return (response.StatusCode == HttpStatusCode.OK);
}
}
catch
{
return false;
}
}
Also when dealing with IDisposables you should take care of releasing all used resources by using the using statement or calling Dispose().
If you are using .NET Framework 4+ you can also use WebRequest.CreateHttp(string) to create your HttpWebRequest.
Putting it simple just use this:
private void button1_Click(object sender, EventArgs e)
{
this.progressBar1.Style = System.Windows.Forms.ProgressBarStyle.Marquee;
Thread thread = new Thread(() => RemoteFileExists("http://www.google.com/"));
thread.IsBackground = true;
thread.SetApartmentState(ApartmentState.STA);
thread.Start();
}
And do the check inside RemoteFileExists .
How can I use HttpWebRequest (.NET, C#) asynchronously?
Use HttpWebRequest.BeginGetResponse()
HttpWebRequest webRequest;
void StartWebRequest()
{
webRequest.BeginGetResponse(new AsyncCallback(FinishWebRequest), null);
}
void FinishWebRequest(IAsyncResult result)
{
webRequest.EndGetResponse(result);
}
The callback function is called when the asynchronous operation is complete. You need to at least call EndGetResponse() from this function.
By far the easiest way is by using TaskFactory.FromAsync from the TPL. It's literally a couple of lines of code when used in conjunction with the new async/await keywords:
var request = WebRequest.Create("http://www.stackoverflow.com");
var response = (HttpWebResponse) await Task.Factory
.FromAsync<WebResponse>(request.BeginGetResponse,
request.EndGetResponse,
null);
Debug.Assert(response.StatusCode == HttpStatusCode.OK);
If you can't use the C#5 compiler then the above can be accomplished using the Task.ContinueWith method:
Task.Factory.FromAsync<WebResponse>(request.BeginGetResponse,
request.EndGetResponse,
null)
.ContinueWith(task =>
{
var response = (HttpWebResponse) task.Result;
Debug.Assert(response.StatusCode == HttpStatusCode.OK);
});
Considering the answer:
HttpWebRequest webRequest;
void StartWebRequest()
{
webRequest.BeginGetResponse(new AsyncCallback(FinishWebRequest), null);
}
void FinishWebRequest(IAsyncResult result)
{
webRequest.EndGetResponse(result);
}
You could send the request pointer or any other object like this:
void StartWebRequest()
{
HttpWebRequest webRequest = ...;
webRequest.BeginGetResponse(new AsyncCallback(FinishWebRequest), webRequest);
}
void FinishWebRequest(IAsyncResult result)
{
HttpWebResponse response = (result.AsyncState as HttpWebRequest).EndGetResponse(result) as HttpWebResponse;
}
Greetings
Everyone so far has been wrong, because BeginGetResponse() does some work on the current thread. From the documentation:
The BeginGetResponse method requires some synchronous setup tasks to
complete (DNS resolution, proxy detection, and TCP socket connection,
for example) before this method becomes asynchronous. As a result,
this method should never be called on a user interface (UI) thread
because it might take considerable time (up to several minutes
depending on network settings) to complete the initial synchronous
setup tasks before an exception for an error is thrown or the method
succeeds.
So to do this right:
void DoWithResponse(HttpWebRequest request, Action<HttpWebResponse> responseAction)
{
Action wrapperAction = () =>
{
request.BeginGetResponse(new AsyncCallback((iar) =>
{
var response = (HttpWebResponse)((HttpWebRequest)iar.AsyncState).EndGetResponse(iar);
responseAction(response);
}), request);
};
wrapperAction.BeginInvoke(new AsyncCallback((iar) =>
{
var action = (Action)iar.AsyncState;
action.EndInvoke(iar);
}), wrapperAction);
}
You can then do what you need to with the response. For example:
HttpWebRequest request;
// init your request...then:
DoWithResponse(request, (response) => {
var body = new StreamReader(response.GetResponseStream()).ReadToEnd();
Console.Write(body);
});
public static async Task<byte[]> GetBytesAsync(string url) {
var request = (HttpWebRequest)WebRequest.Create(url);
using (var response = await request.GetResponseAsync())
using (var content = new MemoryStream())
using (var responseStream = response.GetResponseStream()) {
await responseStream.CopyToAsync(content);
return content.ToArray();
}
}
public static async Task<string> GetStringAsync(string url) {
var bytes = await GetBytesAsync(url);
return Encoding.UTF8.GetString(bytes, 0, bytes.Length);
}
I ended up using BackgroundWorker, it is definitely asynchronous unlike some of the above solutions, it handles returning to the GUI thread for you, and it is very easy to understand.
It is also very easy to handle exceptions, as they end up in the RunWorkerCompleted method, but make sure you read this: Unhandled exceptions in BackgroundWorker
I used WebClient but obviously you could use HttpWebRequest.GetResponse if you wanted.
var worker = new BackgroundWorker();
worker.DoWork += (sender, args) => {
args.Result = new WebClient().DownloadString(settings.test_url);
};
worker.RunWorkerCompleted += (sender, e) => {
if (e.Error != null) {
connectivityLabel.Text = "Error: " + e.Error.Message;
} else {
connectivityLabel.Text = "Connectivity OK";
Log.d("result:" + e.Result);
}
};
connectivityLabel.Text = "Testing Connectivity";
worker.RunWorkerAsync();
.NET has changed since many of these answers were posted, and I'd like to provide a more up-to-date answer. Use an async method to start a Task that will run on a background thread:
private async Task<String> MakeRequestAsync(String url)
{
String responseText = await Task.Run(() =>
{
try
{
HttpWebRequest request = WebRequest.Create(url) as HttpWebRequest;
WebResponse response = request.GetResponse();
Stream responseStream = response.GetResponseStream();
return new StreamReader(responseStream).ReadToEnd();
}
catch (Exception e)
{
Console.WriteLine("Error: " + e.Message);
}
return null;
});
return responseText;
}
To use the async method:
String response = await MakeRequestAsync("http://example.com/");
Update:
This solution does not work for UWP apps which use WebRequest.GetResponseAsync() instead of WebRequest.GetResponse(), and it does not call the Dispose() methods where appropriate. #dragansr has a good alternative solution that addresses these issues.
public void GetResponseAsync (HttpWebRequest request, Action<HttpWebResponse> gotResponse)
{
if (request != null) {
request.BeginGetRequestStream ((r) => {
try { // there's a try/catch here because execution path is different from invokation one, exception here may cause a crash
HttpWebResponse response = request.EndGetResponse (r);
if (gotResponse != null)
gotResponse (response);
} catch (Exception x) {
Console.WriteLine ("Unable to get response for '" + request.RequestUri + "' Err: " + x);
}
}, null);
}
}
Follow up to the #Isak 's answer, which is very good. Nonetheless it's biggest flaw is that it will only call the responseAction if the response has status 200-299. The best way to fix this is:
private void DoWithResponseAsync(HttpWebRequest request, Action<HttpWebResponse> responseAction)
{
Action wrapperAction = () =>
{
request.BeginGetResponse(new AsyncCallback((iar) =>
{
HttpWebResponse response;
try
{
response = (HttpWebResponse)((HttpWebRequest)iar.AsyncState).EndGetResponse(iar);
}
catch (WebException ex)
{
// It needs to be done like this in order to read responses with error status:
response = ex.Response as HttpWebResponse;
}
responseAction(response);
}), request);
};
wrapperAction.BeginInvoke(new AsyncCallback((iar) =>
{
var action = (Action)iar.AsyncState;
action.EndInvoke(iar);
}), wrapperAction);
}
And then as #Isak follows:
HttpWebRequest request;
// init your request...then:
DoWithResponse(request, (response) => {
var body = new StreamReader(response.GetResponseStream()).ReadToEnd();
Console.Write(body);
});
I've been using this for async UWR, hopefully it helps someone
string uri = "http://some.place.online";
using (UnityWebRequest uwr = UnityWebRequest.Get(uri))
{
var asyncOp = uwr.SendWebRequest();
while (asyncOp.isDone == false) await Task.Delay(1000 / 30); // 30 hertz
if(uwr.result == UnityWebRequest.Result.Success) return uwr.downloadHandler.text;
Debug.LogError(uwr.error);
}
Why won't this HTTPHandler work? Both getPageBody and getPageContent await forever and never get back to me. Nothing else happens after the await (used a break point).
Any help would be strongly appreciated!
PS: Visiting the page in the browser does work - so the problem must be on the C# end.
public class HTTPHandler
{
public static async Task<List<String>> getPageBody(String page)
{
WebRequest request = WebRequest.Create(
"http://www.mywebsite.com/dev/api/" + page);
request.Credentials = CredentialCache.DefaultCredentials;
WebResponse response = await request.GetResponseAsync();
Stream dataStream = response.GetResponseStream();
StreamReader reader = new StreamReader(dataStream);
string responseFromServer = reader.ReadToEnd();
//reader.Close();
//response.Close();
return responseFromServer.Split(';').ToList();
}
public static async Task<List<String>> getPageContents(String page)
{
HttpClient client = new HttpClient();
Task<HttpResponseMessage> resp;
await (resp = client.GetAsync("http://www.mywebsite.com/dev/api/" + page)).ContinueWith(
(getTask) =>
{
getTask.Result.EnsureSuccessStatusCode();
});
//HttpResponseMessage resp = await client.GetAsync("http://mywebsite.com/dev/api/" + page);
Task<String> responseBodyAsText = resp.Result.Content.ReadAsStringAsync();
responseBodyAsText.Wait();
return responseBodyAsText.Result.Split(';').ToList();
}
}
I suspect that further up your call stack, you're using Task.Wait or Task.Result.
If you use Wait or Result on async code, you run the risk of deadlock. I explain this in more detail on my blog and in a recent MSDN article.
Hi guys just wondering if somebody could help me try and correctly thread my application, I am constantly hitting an hurdle after another, I have never been to clued up on threading in applications. I have tryed following this http://www.developerfusion.com/code/4654/asynchronous-httpwebrequest/ tutorial.
basically I'm just trying to stop my request from hanging my application
public class Twitter
{
private const string _username = "****",
_password = "****";
private WebResponse webResp;
public string getTimeLine()
{
Thread thread = new Thread(new ThreadStart(TwitterRequestTimeLine));
thread.IsBackground = true;
thread.Start();
using (Stream responseStream = webResp.GetResponseStream())
{
//
using (StreamReader reader = new StreamReader(responseStream))
{
return reader.ReadToEnd();
}
}
}
private void TwitterRequestTimeLine()
{
string aUrl = "http://168.143.162.116/statuses/home_timeline.xml";
HttpWebRequest request = (HttpWebRequest) HttpWebRequest.Create(aUrl);
SetRequestParams(request);
request.Credentials = new NetworkCredential(_username, _password);
//WebResponse tempResp = request.GetResponse();
ThreadState state = new ThreadState();
IAsyncResult result = request.BeginGetResponse(new AsyncCallback(???), ???);
}
private static void SetRequestParams( HttpWebRequest request )
{
request.Timeout = 500000;
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded";
request.UserAgent = "AdverTwitment";
}
}
}
anyone help would be greatly appricated
You really don't need to thread HttpWebRequest.
When you use BeginGetResponse() and EndGetResponse() with HttpWebRequest, it already uses a background thread for you in order to work asynchronously. There is no reason to push this into a background thread.
As for usage: The help for HttpWebRequest.BeginGetResponse demonstrates a complete, asynchronous request.
If this is a WinForms app, the easiest way to keep the GUI responsive while executing the WebRequest is to use a BackgroundWorker component. Drop a BackgroundWorker on your form and call its RunWorkAsync() method. Put the code to execute the WebRequest and read the Response in the DoWork event handler.
Try using an AsyncCallback like Rubens suggested but have the callback call into a separate method to load the data to its destination. If the getTimeline method doesn't return immediately it will cause the application to hang, because the UI Thread is what is running the request itself.
If you use a separate AsyncCallback to be called after the request is done and have it load the data then the method will return immediately and your UI thread can do other things while it waits.
What about this:
private string getTimeLine()
{
string responseValue = "";
string aUrl = "http://168.143.162.116/statuses/home_timeline.xml";
AutoResetEvent syncRequest = new AutoResetEvent(false);
WebRequest request = WebRequest.Create(aUrl);
request.Method = "POST";
request.BeginGetResponse(getResponseResult =>
{
HttpWebResponse response =
(HttpWebResponse)request.EndGetResponse(getResponseResult);
using (StreamReader reader =
new StreamReader(response.GetResponseStream()))
{
responseValue = reader.ReadToEnd();
}
syncRequest.Set();
}, null);
syncRequest.WaitOne();
return responseValue;
}
EDIT: Ok, I tried to keep a method returning a string, that's why I used AutoResetEvent; If you use a BackgroundWorker, you'll get notified when your data is available:
BackgroundWorker worker = new BackgroundWorker();
string responseValue = "";
worker.RunWorkerCompleted += (sender, e) =>
{
// update interface using responseValue variable
};
worker.DoWork += (sender, e) =>
{
string aUrl = "http://168.143.162.116/statuses/home_timeline.xml";
WebRequest request = WebRequest.Create(aUrl);
// .. setup
using(StreamReader reader =
new StreamReader(request.GetResponse().GetResponseStream()))
responseValue = reader.ReadToEnd();
};
worker.RunWorkerAsync();