Why is this implementation of async WebRequests slower than the synchronous implementation? - c#

I have an application that requires many requests to a third party REST service. I thought that modifying this part of the application to make the requests asynchronously would potentially speed things up, so I wrote a POC console application to test things out.
To my surprise the async code takes almost twice as long to complete as the synchronous version. Am I just doing it wrong?
async static void LoadUrlsAsync()
{
var startTime = DateTime.Now;
Console.WriteLine("LoadUrlsAsync Start - {0}", startTime);
var numberOfRequest = 3;
var tasks = new List<Task<string>>();
for (int i = 0; i < numberOfRequest; i++)
{
var request = WebRequest.Create(#"http://www.google.com/images/srpr/logo11w.png") as HttpWebRequest;
request.Method = "GET";
var task = LoadUrlAsync(request);
tasks.Add(task);
}
var results = await Task.WhenAll(tasks);
var stopTime = DateTime.Now;
var duration = (stopTime - startTime);
Console.WriteLine("LoadUrlsAsync Complete - {0}", stopTime);
Console.WriteLine("LoadUrlsAsync Duration - {0}ms", duration);
}
async static Task<string> LoadUrlAsync(WebRequest request)
{
string value = string.Empty;
using (var response = await request.GetResponseAsync())
using (var responseStream = response.GetResponseStream())
using (var reader = new StreamReader(responseStream))
{
value = reader.ReadToEnd();
Console.WriteLine("{0} - Bytes: {1}", request.RequestUri, value.Length);
}
return value;
}
NOTE:
I have also tried setting the maxconnections=100 in the app.config in an attempt to eliminate throttling from the system.net connection pool. This setting doesn't seem to make an impact on the performance.
<system.net>
<connectionManagement>
<add address="*" maxconnection="100" />
</connectionManagement>
</system.net>

First, try to avoid microbenchmarking. When the differences of your code timings are swamped by network conditions, your results lose meaning.
That said, you should set ServicePointManager.DefaultConnectionLimit to int.MaxValue. Also, use end-to-end async methods (i.e., StreamReader.ReadToEndAsync) - or even better, use HttpClient, which was designed for async HTTP.

The async version becomes faster as you increase the number of threads. I'm not certain, however my guess is that you are bypassing the cost of setting up the threads. When you pass this threshold the async version becomes superior. Try 50 or even 500 requests and you should see async is faster. That's how it worked out for me.
500 Async Requests: 11.133 seconds
500 Sync Requests: 18.136 seconds
If you only have ~3 calls then I suggest avoiding async. Here's what I used to test:
public class SeperateClass
{
static int numberOfRequest = 500;
public async static void LoadUrlsAsync()
{
var startTime = DateTime.Now;
Console.WriteLine("LoadUrlsAsync Start - {0}", startTime);
var tasks = new List<Task<string>>();
for (int i = 0; i < numberOfRequest; i++)
{
var request = WebRequest.Create(#"http://www.google.com/images/srpr/logo11w.png") as HttpWebRequest;
request.Method = "GET";
var task = LoadUrlAsync(request);
tasks.Add(task);
}
var results = await Task.WhenAll(tasks);
var stopTime = DateTime.Now;
var duration = (stopTime - startTime);
Console.WriteLine("LoadUrlsAsync Complete - {0}", stopTime);
Console.WriteLine("LoadUrlsAsync Duration - {0}ms", duration);
}
async static Task<string> LoadUrlAsync(WebRequest request)
{
string value = string.Empty;
using (var response = await request.GetResponseAsync())
using (var responseStream = response.GetResponseStream())
using (var reader = new StreamReader(responseStream))
{
value = reader.ReadToEnd();
Console.WriteLine("{0} - Bytes: {1}", request.RequestUri, value.Length);
}
return value;
}
}
public class SeperateClassSync
{
static int numberOfRequest = 500;
public async static void LoadUrlsSync()
{
var startTime = DateTime.Now;
Console.WriteLine("LoadUrlsSync Start - {0}", startTime);
var tasks = new List<Task<string>>();
for (int i = 0; i < numberOfRequest; i++)
{
var request = WebRequest.Create(#"http://www.google.com/images/srpr/logo11w.png") as HttpWebRequest;
request.Method = "GET";
var task = LoadUrlSync(request);
tasks.Add(task);
}
var results = await Task.WhenAll(tasks);
var stopTime = DateTime.Now;
var duration = (stopTime - startTime);
Console.WriteLine("LoadUrlsSync Complete - {0}", stopTime);
Console.WriteLine("LoadUrlsSync Duration - {0}ms", duration);
}
async static Task<string> LoadUrlSync(WebRequest request)
{
string value = string.Empty;
using (var response = request.GetResponse())//Still async FW, just changed to Sync call here
using (var responseStream = response.GetResponseStream())
using (var reader = new StreamReader(responseStream))
{
value = reader.ReadToEnd();
Console.WriteLine("{0} - Bytes: {1}", request.RequestUri, value.Length);
}
return value;
}
}
class Program
{
static void Main(string[] args)
{
SeperateClass.LoadUrlsAsync();
Console.ReadLine();//record result and run again
SeperateClassSync.LoadUrlsSync();
Console.ReadLine();
}
}

In my tests it's faster to use the WebRequest.GetResponseAsync() method for 3 parallel requests.
It should be more noticeable with large requests, many requests (3 is not many), and requests from different websites.
What are the exact results you are getting? In your question you are converting a TimeSpan to a string and calling it milliseconds but you aren't actually calculating the milliseconds. It's displaying the standard TimeSpan.ToString which will show fractions of a second.

It appears that the problem was more of an environmental issue than anything else. Once I moved the code to another machine on a different network, the results were much more inline with my expectations.
The original async code does in fact execute more quickly than the synchronous version. This helps me ensure that I am not introducing additional complexity to our application without the expected performance gains.

Related

Perf benchmark - Delay between send request to API and request received in API

I'm trying to mesure duration process on my API.
For that, I'm using a simple code which call n time the API at the same time.
private HttpClient _httpClient { get; } = new HttpClient();
[TestCase]
public async Task Perf_ExecuteAuthThreads()
{
await Parallel.ForEachAsync(Enumerable.Range(0, NB_THREADS).ToList(), new ParallelOptions
{
MaxDegreeOfParallelism = NB_THREADS
}, async (item, cancellationToken) => await CreateAuthThread(item));
}
private async Task CreateAuthThread(int i)
{
string tokenAuth = string.Empty;
DateTime startDate = DateTime.Now;
var stopWatchGlobal = new Stopwatch();
var stopWatchDetails = new Stopwatch();
stopWatchGlobal.Restart();
try
{
stopWatchDetails.Restart();
using var request = new HttpRequestMessage(HttpMethod.Post, RequestFactory.GetRoute(RequestFactory.RequestType.AUTH));
request.Headers.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
request.Content = JsonContent.Create(RequestFactory.CreateRequest(RequestFactory.RequestType.AUTH));
// Authenticate Method
using HttpResponseMessage response = await _httpClient.SendAsync(request, HttpCompletionOption.ResponseHeadersRead);
response.EnsureSuccessStatusCode();
stopWatchDetails.Stop();
if (response.IsSuccessStatusCode)
{
tokenAuth = (await response.Content.ReadFromJsonAsync<Token>())!.JwtToken;
}
else
{
string responseAsString = await response.Content.ReadAsStringAsync();
//Assert.Fail("Internal server Error : {0}", responseAsString);
}
}
catch { /* DO NOTHING */ }
stopWatchGlobal.Stop();
// Get business transaction Guid
var transactionGuid = new JwtSecurityToken(jwtEncodedString: tokenAuth).Claims.FirstOrDefault(x => x.Type == Constants.JWT_CLAIMS_TRANSACTION_GUID)?.Value ?? null;
// Thread Id;TransactionGuid;Begin Date;End Date;Auth Duration;Business Duration;Result Business;Total Duration;
Console.WriteLine($"\"{i}\";\"{transactionGuid}\";\"{startDate.ToString("MM/dd/yyyy hh:mm:ss.fff tt")}\";\"{DateTime.Now.ToString("MM/dd/yyyy hh:mm:ss.fff tt")}\";\"{stopWatchDetails.ElapsedMilliseconds}\";\"{(tokenAuth != string.Empty ? true : false)}\";\"{stopWatchGlobal.ElapsedMilliseconds}\"");
}
Assuming my local computer and the server have synchronous clock, I notice that a difference about 3 seconds appears between send request to API and receive request by API. This difference don't exist between the response by API and the end of the call on my computer.
I know transport layers cost time, but 3 seconds appears at expensive.
Do you know if is it possible to reduce this time ? I don't think it's "burst time" (meaning : initialization time of communication)...
My API (and my test code) are on .Net 6 and using TLS connection (HTTPS). Request and Response are classic Json, without a lot of datas.

Working with SemaphoreSlim and the ThreadPool for sending HttpWebRequests

I need to understand how SemaphoreSlim works with ThreadPool.
I need to read a file from the local disk of a computer, read each line, and then send that line as a HttpWebRequest to 2 different servers, and correspondingly get 2 HttpWebRequest back.
So lets say that file has 100 requests (this number can be in thousands or even more in a real-time scenario), and when all these requests will be sent I should get back 200 responses (like I mentioned each request should go to 2 different servers and fetch 2 responses from them). Here is my code:
static void Main(string[] args)
{
try
{
SendEntriesInFile(someFileOnTheLocaldisk);
Console.WriteLine();
}
catch (Exception e)
{
Console.WriteLine("Regression Tool Error: Major Unspecified Error:\n" + e);
}
}
}
public class MyParameters
{
}
private void SendEntriesInFile(FileInfo file)
{
static SemaphoreSlim threadSemaphore = new SemaphoreSlim(5, 10);
using (StreamReader reader = file.OpenText())
{
string entry = reader.ReadLine();
while (!String.IsNullOrEmpty(entry))
{
MyParameters myParams = new MyParameters(entry, totalNumberOfEntries, serverAddresses, requestType, fileName);
threadSemaphore.Wait();
ThreadPool.QueueUserWorkItem(new WaitCallback(Send), requestParams);
entry = reader.ReadLine();
}
}
}
private void Send(object MyParameters)
{
MyParameters myParams = (MyParameters)MyParameters;
for(int i=0; i < myParams.ServerAddresses.Count; i++)
{
byte[] bytesArray = null;
bytesArray = Encoding.UTF8.GetBytes(myParams.Request);
HttpWebRequest webRequest = null;
if (reqParams.TypeOfRequest == "tlc")
{
webRequest = (HttpWebRequest)WebRequest.Create(string.Format("http://{0}:{1}{2}", myParams .ServerAddresses[i], port, "/SomeMethod1"));
}
else
{
webRequest = (HttpWebRequest)WebRequest.Create(string.Format("http://{0}:{1}{2}", myParams .ServerAddresses[i], port, "/SomeMethod2"));
}
if (webRequest != null)
{
webRequest.Method = "POST";
webRequest.ContentType = "application/x-www-form-urlencoded";
webRequest.ContentLength = bytesArray.Length;
webRequest.Timeout = responseTimeout;
webRequest.ReadWriteTimeout = transmissionTimeout;
webRequest.ServicePoint.ConnectionLimit = maxConnections;
webRequest.ServicePoint.ConnectionLeaseTimeout = connectionLifeDuration;
webRequest.ServicePoint.MaxIdleTime = maxConnectionIdleTimeout;
webRequest.ServicePoint.UseNagleAlgorithm = nagleAlgorithm;
webRequest.ServicePoint.Expect100Continue = oneHundredContinue;
using (Stream requestStream = webRequest.GetRequestStream())
{
//Write the request through the request stream
requestStream.Write(bytesArray, 0, bytesArray.Length);
requestStream.Flush();
}
string response = "";
using (HttpWebResponse httpWebResponse = (HttpWebResponse)webRequest.GetResponse())
{
if (httpWebResponse != null)
{
using (Stream responseStream = httpWebResponse.GetResponseStream())
{
using (StreamReader stmReader = new StreamReader(responseStream))
{
response = stmReader.ReadToEnd();
string fileName = "";
if(i ==0)
{
fileName = Name is generated through some logic here;
}
else
{
fileName = Name is generated through some logic here;
}
using (StreamWriter writer = new StreamWriter(fileName))
{
writer.WriteLine(response);
}
}
}
}
}
}
}
Console.WriteLine(" Release semaphore: ---- " + threadSemaphore.Release());
}
The only thing is that I'm confused with that when I do something like above my Semaphore allow 5 threads to concurrently execute the Send() method and 5 other threads they wait in a queue for their turn. Since my file contains 100 requests so I should get back 200 responses. Every time I ended up getting only 109 or 107 or 108 responses back. Why I don't get all the 200 responses back. Though using a different code (not discussed here) when I send lets say 10 requests in parallel on 10 different created threads (not using the ThreadPool rather creating threads on-demand) I get back all the 200 responses.
Lots of very helpful comments. To add to them, I would recommend using just pure async io for this.
I would recommend applying the Semaphore as a DelegatingHandler.
You want to register this class.
public class LimitedConcurrentHttpHandler
: DelegatingHandler
{
private readonly SemaphoreSlim _concurrencyLimit = new(8);
public LimitedConcurrentHttpHandler(){ }
public LimitedConcurrentHttpHandler(HttpMessageHandler inner) : base(inner) {}
protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
{
await _concurrencyLimit.Wait(cancellationToken);
try
{
return await base.SendAsync(request, cancellationToken);
}
finally
{
_concurrencyLimit.Release();
}
}
}
This way, you can concentrate on your actual business logic within your actual code.

WebRequest slower in multiply tasks

When I work on the WebRequest inside tasks under Mono, I found that it's very slow to get response from server. But it very fast if sending request out Task. I did some research but there is no solution for it. The code works well on Windows and .NET Core on Linux. So I guess it maybe a problem on Mono. Hope someone can help me out of it.
public static void Main(string[] args)
{
ServicePointManager.DefaultConnectionLimit = 12;
ServicePointManager.Expect100Continue = false;
var tasks = Enumerable.Range(0, 10).Select(async _ =>
{
var timer = Stopwatch.StartNew();
await ValidateUrlAsync("https://www.bing.com");
timer.Stop();
Console.WriteLine(timer.ElapsedMilliseconds);
//request.Abort();
}).ToArray();
Task.WaitAll(tasks);
// Single request
{
var timer = Stopwatch.StartNew();
ValidateUrlAsync("https://www.bing.com").ConfigureAwait(false).GetAwaiter().GetResult();
timer.Stop();
Console.WriteLine(timer.ElapsedMilliseconds);
}
Console.Read();
}
public static async Task<bool> ValidateUrlAsync(string url)
{
using(var response = (HttpWebResponse)await WebRequest.Create(url).GetResponseAsync())
return response.StatusCode == HttpStatusCode.OK;
}
The Mono version: 5.16.0.179.
Ubuntu: 16.04.2 LTS.

Why Backgroundeworker + BlockingCollection Combination is slower?

I have a program accessing database and downloading images. I was using BlockingCollection for that purpose. However, to access some UI elements I decided to use combination of Backgroundworker and BlockingCollection. It reduced speed of processing considerably as compared to speed when only Blockingcollection was used. What can be the reason? Or as I am now accessing UI elements, there is reduction in speed?
Here is the code I am working on:
private void button_Start_Click(object sender, System.EventArgs e)
{
BackgroundWorker bgWorker = new BackgroundWorker();
bgWorker.DoWork += bw_DoWork;
bgWorker.RunWorkerCompleted += bw_RunWorkerCompleted;
bgWorker.ProgressChanged += bw_ProgressChanged;
bgWorker.WorkerSupportsCancellation = true;
bgWorker.WorkerReportsProgress = true;
Button btnSender = (Button)sender;
btnSender.Enabled = false;
bgWorker.RunWorkerAsync();
}
and Do_Work() is as follows:
{
HttpWebRequest request = null;
using (BlockingCollection<ImageFileName> bc = new BlockingCollection<ImageFileName>(30))
{
using (Task task1 = Task.Factory.StartNew(() =>
{
foreach (var fileName in fileNames)
{
string baseUrl = "http://some url";
string url = string.Format(baseUrl, fileName);
request = (HttpWebRequest)WebRequest.Create(url);
request.Method = "GET";
request.ContentType = "application/x-www-form-urlencoded";
var response = (HttpWebResponse)request.GetResponse();
Stream stream = response.GetResponseStream();
img = Image.FromStream(stream);
FileNameImage = new ImageFileName(fileName.ToString(), img);
bc.Add(FileNameImage);
Thread.Sleep(100);
Console.WriteLine("Size of BlockingCollection: {0}", bc.Count);
}
}))
{
using (Task task2 = Task.Factory.StartNew(() =>
{
foreach (ImageFileName imgfilename2 in bc.GetConsumingEnumerable())
{
if (bw.CancellationPending == true)
{
e.Cancel = true;
break;
}
else
{
int numIterations = 4;
Image img2 = imgfilename2.Image;
for (int i = 0; i < numIterations; i++)
{
img2.Save("C:\\path" + imgfilename2.ImageName);
ZoomThumbnail = img2;
ZoomSmall = img2;
ZoomLarge = img2;
ZoomThumbnail = GenerateThumbnail(ZoomThumbnail, 86, false);
ZoomThumbnail.Save("C:\\path" + imgfilename2.ImageName + "_Thumb.jpg");
ZoomThumbnail.Dispose();
ZoomSmall = GenerateThumbnail(ZoomSmall, 400, false);
ZoomSmall.Save("C:\\path" + imgfilename2.ImageName + "_Small.jpg");
ZoomSmall.Dispose();
ZoomLarge = GenerateThumbnail(ZoomLarge, 1200, false);
ZoomLarge.Save("C:\\path" + imgfilename2.ImageName + "_Large.jpg");
ZoomLarge.Dispose();
// progressBar1.BeginInvoke(ProgressBarChange);
int percentComplete = (int)(((i + 1.0) / (double)numIterations) * 100.0);
//if (progressBar1.InvokeRequired)
//{
// BeginInvoke(new MethodInvoker(delegate{bw.ReportProgress(percentComplete)};))
//}
}
Console.WriteLine("This is Take part and size is: {0}", bc.Count);
}
}
}))
Task.WaitAll(task1, task2);
}
}
}
A better option might be to make retrieving the data and writing it to disk run synchronously, and instead use Parallel.ForEach() to allow multiple requests to be in-flight at the same time. That should reduce the amount of waiting in a couple spots:
No need to wait for one HTTP request to complete before issuing subsequent requests.
No need to block on that BlockingCollection
No need to wait for one disk write to complete before firing off the next one.
So perhaps something more like this:
Parallel.ForEach(fileNames,
(name) =>
{
string baseUrl = "http://some url";
string url = string.Format(baseUrl, fileName);
var request = (HttpWebRequest)WebRequest.Create(url);
request.Method = "GET";
request.ContentType = "application/x-www-form-urlencoded";
var response = (HttpWebResponse)request.GetResponse();
Stream stream = response.GetResponseStream();
var img = Image.FromStream(stream);
// Cutting out a lot of steps from the 2nd Task to simplify the example
img.Save(Path.Combine("C:\\path", fileName.ToString()));
});
One possible problem you could run into with this approach is that it will start generating too many requests at once. That might cause resource contention issues, or perhaps the webserver will interpret it as malicious behavior and stop responding to you. You can limit the number of requests that happen at the same time by setting the MaxDegreeOfParallelism. The following example shows how you could limit the operation to process no more than 4 files at the same time.
var options = new ParallelOptions { MaxDegreeOfParallelism = 4 };
Parallel.ForEach(fileNames, (name) => { /* do stuff */ }, options);

a faster way to download multiple files

i need to download about 2 million files from the SEC website. each file has a unique url and is on average 10kB. this is my current implementation:
List<string> urls = new List<string>();
// ... initialize urls ...
WebBrowser browser = new WebBrowser();
foreach (string url in urls)
{
browser.Navigate(url);
while (browser.ReadyState != WebBrowserReadyState.Complete) Application.DoEvents();
StreamReader sr = new StreamReader(browser.DocumentStream);
StreamWriter sw = new StreamWriter(), url.Substring(url.LastIndexOf('/')));
sw.Write(sr.ReadToEnd());
sr.Close();
sw.Close();
}
the projected time is about 12 days... is there a faster way?
Edit: btw, the local file handling takes only 7% of the time
Edit: this is my final implementation:
void Main(void)
{
ServicePointManager.DefaultConnectionLimit = 10000;
List<string> urls = new List<string>();
// ... initialize urls ...
int retries = urls.AsParallel().WithDegreeOfParallelism(8).Sum(arg => downloadFile(arg));
}
public int downloadFile(string url)
{
int retries = 0;
retry:
try
{
HttpWebRequest webrequest = (HttpWebRequest)WebRequest.Create(url);
webrequest.Timeout = 10000;
webrequest.ReadWriteTimeout = 10000;
webrequest.Proxy = null;
webrequest.KeepAlive = false;
webresponse = (HttpWebResponse)webrequest.GetResponse();
using (Stream sr = webrequest.GetResponse().GetResponseStream())
using (FileStream sw = File.Create(url.Substring(url.LastIndexOf('/'))))
{
sr.CopyTo(sw);
}
}
catch (Exception ee)
{
if (ee.Message != "The remote server returned an error: (404) Not Found." && ee.Message != "The remote server returned an error: (403) Forbidden.")
{
if (ee.Message.StartsWith("The operation has timed out") || ee.Message == "Unable to connect to the remote server" || ee.Message.StartsWith("The request was aborted: ") || ee.Message.StartsWith("Unable to read data from the trans­port con­nec­tion: ") || ee.Message == "The remote server returned an error: (408) Request Timeout.") retries++;
else MessageBox.Show(ee.Message, "Error", MessageBoxButtons.OK, MessageBoxIcon.Error);
goto retry;
}
}
return retries;
}
Execute the downloads concurrently instead of sequentially, and set a sensible MaxDegreeOfParallelism otherwise you will try to make too many simultaneous request which will look like a DOS attack:
public static void Main(string[] args)
{
var urls = new List<string>();
Parallel.ForEach(
urls,
new ParallelOptions{MaxDegreeOfParallelism = 10},
DownloadFile);
}
public static void DownloadFile(string url)
{
using(var sr = new StreamReader(HttpWebRequest.Create(url)
.GetResponse().GetResponseStream()))
using(var sw = new StreamWriter(url.Substring(url.LastIndexOf('/'))))
{
sw.Write(sr.ReadToEnd());
}
}
Download files in several threads. Number of threads depends on your throughput. Also, look at WebClient and HttpWebRequest classes. Simple sample:
var list = new[]
{
"http://google.com",
"http://yahoo.com",
"http://stackoverflow.com"
};
var tasks = Parallel.ForEach(list,
s =>
{
using (var client = new WebClient())
{
Console.WriteLine($"starting to download {s}");
string result = client.DownloadString((string)s);
Console.WriteLine($"finished downloading {s}");
}
});
I'd use several threads in parallel, with a WebClient. I recommend setting the max degree of parallelism to the number of threads you want, since unspecified degree of parallelism doesn't work well for long running tasks. I've used 50 parallel downloads in one of my projects without a problem, but depending on the speed of an individual download a much lower might be sufficient.
If you download multiple files in parallel from the same server, you're by default limited to a small number (2 or 4) of parallel downloads. While the http standard specifies such a low limit, many servers don't enforce it. Use ServicePointManager.DefaultConnectionLimit = 10000; to increase the limit.
I think the code from o17t H1H' S'k seems right and all but to perform I/O bound tasks an async method should be used.
Like this:
public static async Task DownloadFileAsync(HttpClient httpClient, string url, string fileToWriteTo)
{
using HttpResponseMessage response = await httpClient.GetAsync(url, HttpCompletionOption.ResponseHeadersRead);
using Stream streamToReadFrom = await response.Content.ReadAsStreamAsync();
using Stream streamToWriteTo = File.Open(fileToWriteTo, FileMode.Create);
await streamToReadFrom.CopyToAsync(streamToWriteTo);
}
Parallel.Foreach is also available with Parallel.ForEachAsync. Parallel.Foreach has a lot of features that the async does't has, but most of them are also depracticed. You can implement an Producer Consumer system with Channel or BlockingCollection to handle the amount of 2 million files. But only if you don't know all URLs at the start.
private static async void StartDownload()
{
(string, string)[] urls = new ValueTuple<string, string>[]{
new ("https://dotnet.microsoft.com", "C:/YoureFile.html"),
new ( "https://www.microsoft.com", "C:/YoureFile1.html"),
new ( "https://stackoverflow.com", "C:/YoureFile2.html")};
var client = new HttpClient();
ParallelOptions options = new() { MaxDegreeOfParallelism = 2 };
await Parallel.ForEachAsync(urls, options, async (url, token) =>
{
await DownloadFileAsync(httpClient, url.Item1, url.Item2);
});
}
Also look into this NuGet Package. The Github Wiki gives examples how to use it. To download 2 million files this is a good library and has also a retry function. To download a file you only have to create an instance of LoadRequest and it downloads it with the name of the file into the Downloads directory.
private static void StartDownload()
{
string[] urls = new string[]{
"https://dotnet.microsoft.com",
"https://www.microsoft.com",
" https://stackoverflow.com"};
foreach (string url in urls)
new LoadRequest(url).Start();
}
I hope this helps to improve the code.

Categories