I have an app that making API requests to the last.fm website using the backgroundWorker. Initially I don't know how many request I'm gonna need to make. The response contains the total number of pages, so I will only get it after the first request. Here is the code below.
private void backgroundWorker_DoWork(object sender, DoWorkEventArgs e)
{
int page = 1;
int totalpages = 1;
while (page <= totalpages)
{
if (backgroundWorker.CancellationPending)
{
e.Cancel = true;
return;
}
//Here is the request part
string Response = RecentTracksRequest(username, from, page);
if (Response.Contains("lfm status=\"ok"))
{
totalpages = Convert.ToInt32(Regex.Match(Response, #"totalPages=.(\d+)").Groups[1].Value);
MatchCollection match = Regex.Matches(Response, "<track>((.|\n)*?)</track>");
foreach (Match m in match)
ParseTrack(m.Groups[1].Value);
}
else
{
MessageBox.Show("Error sending the request.", "Error",
MessageBoxButtons.OK, MessageBoxIcon.Error);
return;
}
if (page >= totalpages)
break;
if (totalpages == 0)
break;
if (page < totalpages)
page++;
}
The problem is that the last.fm API is really slow and it might take up to 5 seconds to get the responce. With a big number of pages, the loading will take a long time.
I want to make parallel requests, say 3 parallel requests at a time. Is it possible? If yes, how can I make it?
Thanks a lot.
You can take advantage of HttpClient, assume you have list of urls:
var client = new HttpClient();
var tasks = urls.Select(url => client.GetAsync(url).ContinueWith(t =>
{
var response = t.Result;
response.EnsureSuccessStatusCode();
//Do something more
}));
If you use async method, you can await all tasks finish like below:
var results = await Task.WhenAll(tasks);
You can do Async Web Request as well, using BeginGetResponse
HttpWebRequest webRequest;
webRequest.BeginGetResponse(new AsyncCallback(callbackfunc), null);
void callbackfunc(IAsyncResult response)
{
webRequest.EndGetResponse(response);
}
Related
I am implementing a run() function that can not be async. The goal is to send a get request to a server, and then download some files based on the result of the get request and return the number of files downloaded. I have written an async function to do that but I want to essentially "await" it before the rest of my main function continues. I am unable to achieve this behavior currently as the function just hangs. Not sure why its hanging :(
I think I just need some insights into Task and why this isn't working as expected. I am familiar with promises in JS so I thought this wouldn't be that difficult.
Thank you!
public int run(){
FilesManager test = new FilesManager();
string path = Path.Combine("C:\Users\username\Documents", "Temp");
Task<int> T_count = test.Downloadfiles(path); //TODO: trying to "await" this before the messageBoxes
Task.WaitAll(T_count);
int count = T_count.Result;
MessageBox.Show("File Downloaded");
MessageBox.Show(count.ToString());
}
public async Task<int> Downloadfiles(string path)
{
String[] response = await getClient.GetFromJsonAsync<string[]>("http://localhost:3000");
int counter = 0;
try
{
foreach (string url in response)
{
Uri uri = new Uri(url);
var response2 = await getClient.GetAsync(uri);
using (var fs = new FileStream(
path + counter.ToString(),
FileMode.Create))
{
await response2.Content.CopyToAsync(fs);
}
counter++;
}
return counter;
}catch(Exception e)
{
while (e != null)
{
MessageBox.Show(e.Message);
e = e.InnerException;
}
return 0;
}
}
EDIT:
Still not able to get the task.WaitAll(T_count) to work. With some more debugging, the execution jumps from the response2 = await getClient.GetAsync... straight into the waitAll, never hitting the copyToAsync or counter++.
Sync-over-async is a fundamentally difficult problem, because you need to guarantee that continuations never try to run on the thread you are blocking on, otherwise you will get a deadlock as you have seen. Ideally you would never block on async code, but sometimes that is not possible.
Task.Run(...)..GetAwaiter().GetResult() is normally fine to use for this purpose, although there are still some circumstances when it can deadlock.
Do not call the UI from inside the async function, therefore you must move the catch with MessageBox.Show to the outer function.
You can also make this more efficient, by using HttpCompletionOption.ResponseHeadersRead, and you are missing a using on the response2 object.
public int run()
{
FilesManager test = new FilesManager();
string path = Path.Combine("C:\Users\username\Documents", "Temp");
try
{
int count = Task.Run(() => test.Downloadfiles(path)).GetAwaiter().GetResult();
MessageBox.Show("File Downloaded");
MessageBox.Show(count.ToString());
return count;
}
catch(Exception e)
{
while (e != null)
{
MessageBox.Show(e.Message);
e = e.InnerException;
}
return 0;
}
}
public async Task<int> Downloadfiles(string path)
{
String[] response = await getClient.GetFromJsonAsync<string[]>("http://localhost:3000");
int counter = 0;
try
{
foreach (string url in response)
{
using (var response2 = await getClient.GetAsync(url, HttpCompletionOption.ResponseHeadersRead))
using (var fs = new FileStream(
path + counter.ToString(),
FileMode.Create))
{
await response2.Content.CopyToAsync(fs);
}
counter++;
}
return counter;
}
}
Another option is to remove and afterwards restore the SynchronizationContext, as shown in this answer.
I am trying to do multiple web requests from websites that take a long time to load the full response (really long log files). I want to implement a method that will continue to create web requests while waiting for the response of the other ones and speed the process.
Tried to implement an async- await method but I don't think is creating simultaneous calls and it's instead running synchronously. I am very confused in how to implement the async await, I looked at other similar questions but still not very clear, how could I fix this code to run async?
This is my code:
public static async Task<String> MakeRequestAsync(String url)
{
String responseText = null;
try
{
HttpWebRequest request = WebRequest.Create(url) as HttpWebRequest;
request.Proxy = null;
WebResponse response = request.GetResponse();
Stream responseStream = response.GetResponseStream();
string sr = new StreamReader(responseStream).ReadToEnd();
response.Close();
responseStream.Close();
responseText = sr;
}
catch (Exception e)
{
Console.WriteLine("Error: " + e.Message);
}
return responseText;
}
..
private void button_test_Click(object sender, EventArgs e)
{
for (int n = 0; n < 5; n++) //multiple requests
{
String a = AsyncTest().Result;
}
}
private async Task<String> AsyncTest()
{
String b = await MakeRequestAsync(url);
return b;
}
Also any suggestions on how could I speed this process would be great.
None of your IO calls are async. WebRequest has no async/await pattern either, you can get async behaviour using BeginGetResponse with AsyncCallback but there is a better way to go. Use HttpClient rather, it is the recommend way to make http calls.
I.e
public static async Task<String> MakeRequestAsync(string url)
{
using(var client = new HttpClient()) // don't do this, inject it as a singleton
{
using (var response = await client.GetAsync(url)
return await response.Content.ReadAsStringAsync();
}
}
and then change your Click handler to be an async void:
private async void button_test_Click(object sender, EventArgs e)
{
var tasks = new List<Task<string>>();
for (int n = 0; n < 5; n++) //multiple requests
{
tasks.Add(MakeRequestAsync(url));
}
await Task.WhenAll(tasks); // wait for all of them to complete;
foreach(var task in tasks)
{
var str = await task; // or task.Result, won't block, already completed;
}
}
You should use Microsoft's Reactive Framework (aka Rx) - NuGet System.Reactive and add using System.Reactive.Linq; - then you can do this:
var query =
from n in Observable.Range(0, 5)
from r in Observable.Using(
() => new WebClient(),
wc => Observable.Start(() => wc.DownloadString($"{url}{n}")))
select new { n, r };
IDisposable subscription =
query
.ToArray()
.Select(xs => xs.OrderBy(x => x.n).Select(x => x.r).ToArray())
.Subscribe(rs =>
{
/* Do something with the results */
});
It's all multi-threaded and non-blocking.
If you don't care about the order in which the results are returned then just do this:
var query =
from n in Observable.Range(0, 5)
from r in Observable.Using(
() => new WebClient(),
wc => Observable.Start(() => wc.DownloadString($"{url}{n}")))
select r;
IDisposable subscription =
query
.Subscribe(r =>
{
/* Do something with each result */
});
Calling subscription.Dispose() will stop the query if you need to cancel it.
You can write your MakeRequestAsync method like this:
public static async Task<String> MakeRequestAsync(String url)
{
return await
Observable.Using(
() => new WebClient(),
wc => Observable.Start(() => wc.DownloadString(url)));
}
I have this code to download data from multiple websites.
I need to run this code for about 50.000 times.
However after running the code for 2 minutes (about 4000 times) I get the TaskCanceledException and my CPU goes to 100% and my process slows down incredibly.
This is the code:
public async Task<string[]> GetDataAsync(string address, string postalCode)
{
var path = $"{address} {postalCode}/"; // build proper path for request
var textFrom1 = "";
string textFrom3 = "";
string textFrom2 = "";
while (true)
{
try
{
textFrom1 = await client.GetStringAsync("http://website1.com/" + path);
break;
}
catch (Exception e) //404
{
await Task.Delay(10000); // try every 10 seconds (so I do not bomb the server with requests).
}
}
if (textFrom1.Split(':', ',')[1] == "0")
{
return new string[] { "null", "null", "null", "null", "null" };
}
while (true)
{
try
{
textFrom2 = await client.GetStringAsync("http://website2.com/" + textFrom1.Split('"', '"')[11]);
break;
}
catch (Exception e)
{
await Task.Delay(10000);
}
}
while (true)
{
try
{
textFrom3 = await client.GetStringAsync("http://website3.com/" + textFrom2.Split('"', '"')[3]);
break;
}
catch (Exception e)
{
await Task.Delay(10000);
}
}
var allData = await Task.Run(() => JsonConvert.DeserializeObject<List<RootObject>>(textFrom3));
var item = allData.First();
return new string[] { item.item1, item.item2, item.item3, item.item4, item.item5 };
}
And this is the way I create my tasks:
public string[][] GetAll(
IEnumerable<string> adresses,
IEnumerable<string> postalCodes)
{
// Start all tasks one by one without waiting for responses
var tasks = adresses.Zip(postalCodes, (addr, code) => { return GetDataAsync(addr, code); });
return Task.WhenAll(tasks).Result;
}
Is there any way I can optimize my code, so that I do not have this exception and therefor not slowing down my process?
I hope someone can help me, thank you!
I think you are reaching the maximum number of tasks windows can handle at once. You could split your 50.000 requests to smaller groups or write some code to handle your running async-tasks to make sure that you don't start too much tasks at once.
Furthermore you probably reach the maximum number of ports. Everytime starting a new connection your application will open a new dynamic port.
I have to send 100,000 http requests every few seconds and with the code provided below it takes 19 seconds to send the requests and its longer than my interval.
During this time the CPU usage and Ethernet usage is 100% . I have tried it in a dual processor computer with a higher band width but the same result.
Is there any other way to have a better performance.
protected async void btnStartCallWhenAll_Click(object sender, EventArgs e)
{
t1 = DateTime.Now;
// Make a list of web addresses.
List<string> urlList = SetUpURLList(Convert.ToInt32(txtNoRecordsToAdd.Text));
// One-step async call.
await ProcessAllURLSAsync(urlList);
t2 = DateTime.Now;
double spent = t2.Subtract(t1).TotalMilliseconds;
txtTimeElapsed.Text = " Time Spent :" + spent.ToString() ;
}
private List<string> SetUpURLList(int No)
{
List<string> urls = new List<string>
{
};
for (int i = 1; i <= No; i++)
urls.Add("http://msdn.microsoft.com/library/windows/apps/br211380.aspx");
return urls;
}
private async Task ProcessAllURLSAsync(List<string> urlList)
{
ServicePointManager.UseNagleAlgorithm = true;
ServicePointManager.Expect100Continue = true;
ServicePointManager.CheckCertificateRevocationList = true;
ServicePointManager.MaxServicePointIdleTime = 10000;
ServicePointManager.DefaultConnectionLimit = 1000;
IEnumerable<Task> CallingTasksQuery =
from url in urlList select CallURLAsync(url);
Task[] CallingTasks = CallingTasksQuery.ToArray();
await Task.WhenAll(CallingTasks);
}
private async Task CallURLAsync(string url)
{
var content = new MemoryStream();
var webReq = (HttpWebRequest)WebRequest.Create(url);
using (WebResponse response = await webReq.GetResponseAsync())
{
}
}
Referring to HttpWebRequest is extremely slow!
Ensure you've set your:
webReq.Proxy = null;
It appears as though the startup of WebRequest will look for the default proxy.
I am using webrequest to fetch some image data. The url may be invaild sometime. In case of invalid URL, begingetresponse is taking time equals to timeout period. Also the control become unresponsive during that period. In other word the async callback is not working asynchronously. Is this expected behaviour?
try
{
// Async requests
WebRequest request = WebRequest.Create(uri);
request.Timeout = RequestTimeOut;
RequestObject requestObject = new RequestObject();
requestObject.Request = request;
request.BeginGetResponse(this.ProcessImage, requestObject);
}
catch (Exception)
{
ShowErrorMessage(uri);
}
private void ProcessImage(IAsyncResult asyncResult)
{
try
{
RequestObject requestObject = (RequestObject)asyncResult.AsyncState;
WebRequest request = requestObject.Request;
WebResponse response = request.EndGetResponse(asyncResult);
Bitmap tile = new Bitmap(response.GetResponseStream());
// do something
}
catch (Exception)
{
ShowErrorMessage();
}
}
looks like this is an issue with .NET. BeginGetResponse blocks until DNS is resolved. In case of wrong URL (like http://somecrap) it tries until it gets timeout. See the following links -
link1 and link2
I just ran into this same situation. While it's not a perfect workaround I decided to use the Ping.SendAsync() to ping the site first. Good part is the async part return immediately. Bad part is the extra step AND not all sites respond to Ping requests.
public void Start(WatchArgs args)
{
var p = new System.Net.NetworkInformation.Ping();
args.State = p;
var po = new System.Net.NetworkInformation.PingOptions(10, true);
p.PingCompleted += new PingCompletedEventHandler(PingResponseReceived);
p.SendAsync(args.Machine.Name, 5 * 1000, Encoding.ASCII.GetBytes("watchdog"), po, args);
}
private void PingResponseReceived(object sender, .PingCompletedEventArgs e)
{
WatchArgs args = e.UserState as WatchArgs;
var p = args.State as System.Net.NetworkInformation.Ping;
p.PingCompleted -= new System.Net.NetworkInformation.PingCompletedEventHandler(HttpSmokeWatcher.PingResponseReceived);
args.State = null;
if (System.Net.NetworkInformation.IPStatus.Success == e.Reply.Status)
{
// ... BeginGetResponse now
}
else
{
/// ... machine not available
}
}
Just code and running for a day but initial result look promising.