I have a windows form application that reads a bunch of urls(~800) from a text file to List. The application then shows the status code of all the urls.
The problem is If I run a normal for loop from 0 to list count, it takes a lot of time. I need to speed up the process to the maximum possible without blocking the UI. Below is my code
private async void button1_Click(object sender, EventArgs e)
{
System.IO.StreamReader file = new System.IO.StreamReader("urls.txt");
while ((line = file.ReadLine()) != null)
{
pages.Add(line);
}
file.Close();
for(int i=0; i<pages.Count; i++)
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(pages[i]);
int code = 0;
try
{
WebResponse response = await request.GetResponseAsync();
HttpWebResponse r = (HttpWebResponse)response;
code = (int)r.StatusCode;
}
catch (WebException we)
{
var r = ((HttpWebResponse)we.Response).StatusCode;
code = (int)r;
}
}
//add the url and status code to a datagridview
}
One approach would be to use tasks, so that you are not waiting for the last request to complete, before starting the next.
Task<int> tasks;
for (int i = 0; i < 10; i++)
{
tasks = Task.Run<int>(() =>
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(pages[i]);
int code = 0;
try
{
WebResponse response = request.GetResponse();
HttpWebResponse r = (HttpWebResponse)response;
code = (int)r.StatusCode;
}
catch (WebException we)
{
var r = ((HttpWebResponse)we.Response).StatusCode;
code = (int)r;
}
return code;
}
);
}
await tasks;
Related
I need to download numerous web pages' sources. So I need to do that as fast as possible. Here is my codes.
private static async Task<string> downloadsource(string link)
{
ServicePointManager.Expect100Continue = false;
WebRequest req = WebRequest.Create(link);
req.Proxy = null;
req.Method = "GET";
WebResponse res = await siteyeBaglantiTalebi.GetResponseAsync();
StreamReader read = new StreamReader(res.GetResponseStream());
return read.ReadToEnd();
}
List<string> links = new List<string>(){... including some web page links};
private static List<string> source_list(List<string> links)
{
List<string> sources = new List<string>();
for (int i = 0; i < links.Count; i++)
{
Task<string> _task = downloadsource(links[i]);
Console.WriteLine("Downloaded : " + i);
sources.Add(_task.Result);
}
return sources;
}
I was wondering if this code is the fastest way or it can be enhanced.
Can u pls help me with that ?
You are performing a _task.Result call inside each loop. Your code will run as fast as if you downloaded each page one after another if you code it like that.
Try this instead:
private async static Task<List<string>> source_list(List<string> links)
{
List<Task<string>> sources = new List<Task<string>>();
for (int i = 0; i < links.Count; i++)
{
Task<string> _task = downloadsource(links[i]);
Console.WriteLine("Downloading : " + i);
sources.Add(_task);
}
return (await Task.WhenAll(sources)).ToList();
}
This would be even better:
private async static Task<string[]> source_list(List<string> links)
{
return await Task.WhenAll(links.Select(l => downloadsource(l)));
}
Also I cleaned up your downloadsource method:
private static async Task<string> downloadsource(string link)
{
ServicePointManager.Expect100Continue = false;
WebRequest req = WebRequest.Create(link);
req.Proxy = null;
req.Method = "GET";
using (WebResponse res = await req.GetResponseAsync())
{
using (StreamReader read = new StreamReader(res.GetResponseStream()))
{
return read.ReadToEnd();
}
}
}
my environment version is 2.0, I am currently working with Unity 5.5, c#.
I am experiencing a stall the 3rd time i try to get an HttpWebRequest request stream.
What I want to achieve is having a mechanism of "retry": when I send a request I wait for the response and if it times out I send another request and refresh the time out. I stop this loop when I hit MAX_ATTEMPTS or one of the requests get a response (even if it is from a request that has timed out).
The code that handles this loop is here:
IEnumerator DoRequest()
{
List<IAsyncResult> results = new List<IAsyncResult>(MAX_ATTEMPTS);
int attemptNumber = 1;
while (attemptNumber <= MAX_ATTEMPTS)
{
IAsyncResult result = null;
try
{
result = SendRequest();
}
catch (Exception e)
{
ProcessException(e);
yield break;
}
if (result != null)
{
results.Add(result);
DateTime timeOutTime = DateTime.UtcNow + TIMEOUT;
yield return null;
while (DateTime.UtcNow < timeOutTime)
{
int completedIndex = -1;
if (IsAnyResultCompleted(results, out completedIndex) == true)
{
ProcessResponse(results[completedIndex]);
yield break;
}
else
{
yield return null;
}
}
}
attemptNumber++;
}
if (results.Count != 0)
{
ProcessResponseFail("Timed out");
}
else
{
ProcessResponseFail("Unable to generate a request");
}
}
Where TIMEOUT is a TimeSpan of 10 seconds. And IsAnyResultCompleted is just a for loop that checks if any request has IsCompleted set to true.
Here I create and send the request:
IAsyncResult SendRequest()
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(RequestActionPath);
request.ContentType = "application/json";
request.Accept = "application/json";
request.Method = "POST";
string json = JsonUtility.ToJson(_dependency);
using (StreamWriter sw = new StreamWriter(request.GetRequestStream()))
{
sw.Write(json);
sw.Flush();
}
IAsyncResult result = request.BeginGetResponse(new AsyncCallback(onResponse), request);
return result;
}
"onResponse" is a cached delegate and it is assigned to this:
void OnResponse(IAsyncResult asynchronousResult)
{
try
{
HttpWebRequest request = (HttpWebRequest)asynchronousResult.AsyncState;
request.EndGetResponse(asynchronousResult);
}
catch (WebException e)
{
string responseStream;
using (var streamReader = new StreamReader(e.Response.GetResponseStream()))
{
responseStream = streamReader.ReadToEnd();
}
Utility.Console.LogError(responseStream);
}
}
I am well aware of this and I also tried to set ServicePointManager.DefaultConnectionLimit = 1000000 but the result is always the same, after 2 sent requests GetRequestStream() in SendRequest stalls for 5 seconds and I cannot understand why.
I have almost 200 text files in my laptop, I wrote the code in C# which reads these text files line by line and makes a directory per each line in FTP server.
This is my code:
static void Main()
{
for (int i = 0; i <= 200; i++)
{
var lines = File.ReadAllLines(#"D:\file_" + i.ToString().PadLeft(5, '0') + ".txt");
foreach (var line in lines)
{
try
{
WebRequest request = WebRequest.Create("ftp://myftp/dir/" + line);
request.Method = WebRequestMethods.Ftp.MakeDirectory;
request.Credentials = new NetworkCredential("user", "pass");
request.GetResponse();
}
catch (Exception ex)
{}
}
}
}
But this is very slow to create directories, are there other faster ways to do this? For example, get text file as an array and than create all of its directories.
Reading of the text file is really not the problem. The slow part is the FTP.
Use more threads to parallelize the processing:
List<Task> tasks = new List<Task>();
for (int i = 0; i <= 200; i++)
{
tasks.Add(new Task(() =>
{
var lines =
File.ReadAllLines(#"D:\file_" + i.ToString().PadLeft(5, '0') + ".txt");
foreach (var line in lines)
{
WebRequest request = WebRequest.Create("ftp://myftp/dir/" + line);
request.Method = WebRequestMethods.Ftp.MakeDirectory;
request.Credentials = new NetworkCredential("user", "pass");
request.GetResponse();
}
}
));
}
Task.WaitAll(tasks.ToArray());
Though note that you should also take care of disposing the WebResponse's.
The slow part of your programm are the requests you are sending one by one.
You can do some tricks to speed them up:
// allow more connections at a time
ServicePointManager.DefaultConnectionLimit = 30;
// don't wait the 100ms every request do
ServicePointManager.Expect100Continue = false;
Further you can send the request in multi-threading, so you don't have to wait for every request until it's finished. But be aware, that a lot of request can bring down a server. 200 shouldn't be a problem.
Here you have some code you could test:
static void Main()
{
// allow more connections at a time
ServicePointManager.DefaultConnectionLimit = 30;
// don't wait the 100ms every request do
ServicePointManager.Expect100Continue = false;
List<Task> tasks = new List<Task>();
for (int i = 0; i <= 200; i++)
{
var lines = File.ReadAllLines(#"D:\file_" + i.ToString().PadLeft(5, '0') + ".txt");
foreach (var line in lines)
{
tasks.Add(Task.Run(() =>
{
try
{
WebRequest request = WebRequest.Create("ftp://myftp/dir/" + line);
request.Method = WebRequestMethods.Ftp.MakeDirectory;
request.Credentials = new NetworkCredential("user", "pass");
request.GetResponse();
}
catch (Exception ex)
{ }
}
));
}
}
Task.WaitAll(tasks.ToArray());
}
I am having a hard time convert the below code which i have created in 4.0 to 4.5 using HttpClient.
According to my understand i guess if i create multiple web requests in the GUI thread itself without blocking the GUI if i got with asynchronous requeest.
how to convert the below code to Asynchronous using HttpClient in 4.5
// This is what called when button is clicked
Task t3 = new Task(SpawnTask);
t3.Start();
//if noofthreads are less 50 then GUI is woking fine.. if number increases then takes much time for repaint..
//where as other softwares are working without any problem even if the threads are more than 500!! in the same system
public void SpawnTask()
{
try
{
ParallelOptions po = new ParallelOptions();
po.CancellationToken = cts.Token;
po.MaxDegreeOfParallelism = noofthreads;
Parallel.ForEach(
urls,
po,
url => checkpl(url));
}
catch (Exception ex)
{
}
}
public void checkpl(string url)
{
try
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.Timeout = 60*1000;
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
string stext = "";
using (BufferedStream buffer = new BufferedStream(response.GetResponseStream()))
{
using (StreamReader reader = new StreamReader(buffer))
{
stext = reader.ReadToEnd();
}
}
response.Close();
if (stext .IndexOf("domainname.com") != -1)
{
tfound = tfound + 1;
string lext = "Total Found : "+tfound.ToString();
label3.BeginInvoke(new InvokeDelegate(UpdateLabel), ltext);
slist.Add(url);
textBox2.BeginInvoke(new InvokeDelegate4(UpdateText), "Working Url " + url);
}
}
catch (Exception ex)
{
}
}
Since you are using .NET 4.5 you can use the new async and await keywords. Here is what it might look like.
private async void YourButton_Click(object sender, EventArgs args)
{
YourButton.Enabled = false;
try
{
var tasks = new List<Task>();
foreach (string url in Urls)
{
tasks.Add(CheckAsync(url));
}
await TaskEx.WhenAll(tasks);
}
finally
{
YourButton.Enabled = true;
}
}
private async Task CheckAsync(string url)
{
bool found = await UrlResponseContainsAsync(url, "domainname.com");
if (found)
{
slist.Add(url);
label3.Text = "Total Found: " + slist.Count.ToString();
textbox2.Text = "Working Url " + url;
}
}
private async Task<bool> UrlResponseContainsAsync(string url, string find)
{
var request = WebRequest.Create(url);
request.Timeout = 60 * 1000;
using (WebResponse response = await request.GetResponseAsync())
{
using (var buffer = new BufferedStream(response.GetResponseStream()))
using (var reader = new StreamReader(buffer))
{
string text = reader.ReadToEnd();
return text.Contains(find);
}
}
}
I want to stress my website with multiple access. To do that i created a windows based application that call 1000 times the website.
Unfortunatly it work just for 2 call. This is the code:
static void myMethod( int i)
{
int j = 0;
try
{
string url = "";
WebRequest wr = null;
HttpWebResponse response = null;
url = String.Format("http://www.google.com");
wr = WebRequest.Create(url);
//wr.Timeout = 1000;
response = (HttpWebResponse)wr.GetResponse();
MessageBox.Show("end");
}
catch (Exception ex)
{
MessageBox.Show(j.ToString() + " " + ex.Message);
}
}
private void button1_Click(object sender, EventArgs e)
{
for (int i = 0; i < 1000; i++)
{
ThreadStart starter = delegate { myMethod(i); };
Thread thread = new Thread(starter);
thread.Start();
}
}
Rather use the Free WCAT Tool to load test your ASP.NET page.
Also view this video [How Do I:] Load Test a Web Application
If you have Visual Studio 2010 Ultimate, see this link
I hope this helps.
By default HttpRequest only allows two connections to the same host.
You can change this by setting the DefaultConnectionLimit property.
Try disposing the IDisposable instances (i.e. the response) before continuing.
static void myMethod( int i)
{
int j = 0;
try
{
string url = String.Format("http://www.google.com");
WebRequest wr = WebRequest.Create(url);
using(HttpWebResponse response = (HttpWebResponse)wr.GetResponse())
using(Stream responseStream = wr.GetResponseStream())
{
//handle response / response stream
}
MessageBox.Show("end"); //this won't scale!!!
}
catch (Exception ex)
{
MessageBox.Show(j.ToString() + " " + ex.Message);
}
}