I have a http server and I need to download files from that server to my computer every time I launch the app on my PC, I need to download about five thousand files and each of them are about 1-2 kb. Here is the code that I use for it:
WebClient[][] wbc = new WebClient[1][];
for(int file=0 ; file < myfilecount ; file++)
{
wbc[0][film] = new WebClient();
wbc[0][film].Credentials = new NetworkCredential(username, password);
wbc[0][film].DownloadFileCompleted += Form4_DownloadFileCompleted;
wbc[0][film].DownloadFileTaskAsync("http://MYIPADDRESS/File" + file.ToString(), databaselocation + "\\File" + file.toString());
}
When I do this it downloads files into ram in about 3 sec. But it takes about one minute to write them to my hdd. Is there any faster way to download those files to my HDD?
Also I am getting the information about the count of those files by a file that I write, so is there any better way to download all of them?
I agree if this is required every time you launch the app on your pc you should rethink the process. But then again, I don't completely know your circumstances. That aside, you can make a queued collection of request tasks and wait for them all concurrently:
var requestTasks = Enumerable.Range(0, myFileCount).Select(i => {
var webClient = new WebClient
{
Credentials = new NetworkCredential(username, password),
DownloadFileCompleted += Form4_DownloadFileCompleted
};
return webClient.DownloadFileTaskAsync("http://MYIPADDRESS/File" + i.ToString(), Path.Combine(databaselocation, "file" + i.ToString()));
});
await Task.WhenAll(requestTasks);
Also I believe WebClient is limited to the number of requests that can be made concurrently to the same host, you can configure this by using ServicePointManager:
ServicePointManager.DefaultConnectionLimit = carefullyCalculatedNumber;
I'd be careful with the number of connections allowed, too high of a limit can also become a problem. Hope this helps.
Related
I'm using the transferUtility class to upload files into S3 using input stream in .NET. The problem is, uploading a file of 4MB takes around a minute. I tried both transferUtility.Upload and S3Client.PutObject and the upload time didn't seem to change. The code is below:
WebRequest.DefaultWebProxy = null;
this.S3Client = new Amazon.S3.AmazonS3Client(accessKeyId, secretAccessKey, endPoint);
this.transferUtility = new TransferUtility(this.S3Client);
Amazon.S3.Transfer.TransferUtilityUploadRequest transferRequest = new TransferUtilityUploadRequest();
transferRequest.BucketName = s3Bucket;
transferRequest.CannedACL = Amazon.S3.S3CannedACL.PublicRead;
transferRequest.InputStream = (Stream)fileStream;
transferRequest.Key = newFileName;
transferRequest.Headers.Expires = new DateTime(2020, 1, 1);
this.transferUtility.Upload(transferRequest);
Any help would be appreciated.
Thank you,
I found that this is happening because of an uploading the file stream to the application server takes a very long time.
This is likely also due to the C# SDK MD5sum'ing every chunk it uploads (CPU performance hit and thus slows down the upload while an MD5Sum is computed).
See https://github.com/aws/aws-sdk-net/issues/832#issuecomment-383756520 for workaround.
I am new to this site so I apologise in advance if I made any formatting errors. Let me know if further clarification is needed.
I am currently working on a program in C# which aims at downloading files from several http or ftp sources.
I want it to go through several IP addresses and then check if several methods could download a known file successful. If one of these methods could download the file successfully it should go to the next IP and do the same thing again.
So far I am using a foreach loop that runs through an array containing the IP and creating a folder named "IPxx" and a FTP and HTTP URI because I do not know in advance which IP needs a FTP or HTTP address:
string[] ipArray = new string[4];
ipArray[0]= "xxx.xxx.xxx.xx";
ipArray[1]= "xxx.xxx.xxx.xx";
ipArray[2]= "xxx.xxx.xxx.xx";
ipArray[3]= "xxx.xxx.xxx.xx";
foreach(string ip in ipArray )
{
string ipXx = "IP" + ip.Substring(ip.Length-2);
string ipOrdner = folder + #"\" + ipXx;
Directory.CreateDirectory(ipOrdner);
string ftpAddr= "ftp://" + ip;
string httpAddr= "http://"+ip;
//DownloadTestMethod(httpAddr, ipFolder);
//DownloadTestMethod2(ftpAddr, ipFolder);
//DownloadTestMethod3(htppAddr, ipFolder);
}
So far so good, everything up to this level is working as expected.
However I struggle as soon as I need to go through several Download Methods and check if I can download the file successfully and if not it should go to the next DownloadMethod and try the same.
I came up with following DownloadTestMethod:
public static void DownloadTestMethod(string httpAddr, string ipFolder)
{
string fileName = "test_10k.bin";
string downloadpath = httpAddr + "/" + fileName;
// I want it to check if the http site is online/working
WebRequest request = WebRequest.Create(downloadpath);
request.Proxy.Credentials = CredentialCache.DefaultCredentials;
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
if (response != null)
{
WebClient webClient = new WebClient();
webClient.Proxy.Credentials=CredentialCache.DefaultCredentials;
webClient.DownloadFile(downloadpath, ipFolder + #"\" + fileName);
}
}
It is working for a successfully downloaded file, however I seem to get "stuck" in this method if the method cannot downloaded the requested file.
In this case I want it to jump to the next DownloadMethod and check again until all methods are checked.
Furthermore, if the program went through every IP and it could not download anytrhing, I want it to send an automated email. I know how to implement the EmailMethod but once again I struggle with flow control.
I am an absolute beginner and have no idea how to go from here to get my desired flow.
You might want to look at Task Parallel Library and in particular, its WhenAll() method.
Under it, you might want to look at error handling for the case that any of your methods return error / raise exception.
https://msdn.microsoft.com/en-us/library/dd270695(v=vs.110).aspx
I am using BackgroundTransferRequest to download files, I have more than 6,500 mp3 that user can download them at once clicking on download all button Or user can download individual file.
I cannot add more than 25 files in download BackgroundTransferRequest. What is the workaround with it to add more than 25 files in download queue.
When it reaches the level, the exception is
Unable to download. The application request limit has been reached
Code for adding in Queue, After all files are added. I am processing download.
transferFileName = aya.DownloadUri;
Uri transferUri = new Uri(Uri.EscapeUriString(aya.DownloadUri), UriKind.RelativeOrAbsolute);
BackgroundTransferRequest transferRequest = new BackgroundTransferRequest(transferUri);
transferRequest.Method = "GET";
string downloadFile = transferFileName.Substring(transferFileName.LastIndexOf("/") + 1);
Uri downloadUri = new Uri(downloadLocation + aya.ChapterID + "/" + downloadFile, UriKind.RelativeOrAbsolute);
transferRequest.DownloadLocation = downloadUri;
transferRequest.Tag = string.Format("{0},{1},{2}", downloadFile, aya.ID, aya.ChapterID);
transferRequest.TransferPreferences = TransferPreferences.AllowBattery;
BackgroundTransferService.Add(transferRequest);
You must attach an event handler to BackgroundTransferRequest.StatusChanged. On appropriate state you must explicitly remove transfer from BackgroundTransferService. As you may now, the requests have to be removed from BackgroundTransferService manually. All that is explained in detail in introduction to background transfers on msdn.
You should create a queue of files to download, start with placing first 25 transfers in BackgroundTransferService and after the BackgroudTransferService.Remove(..), you can start next transfer from your queue.
I know a few similar questions have been asked on how to download files using WebClient. I can download individual files perfectly fine, but I want to download a range of files. Anywhere from 1-6000 files. I can download them just fine into my current directory, but I am stumped on how to download them to a different directory based upon where they're being downloaded from. Do I need to temporarily change the current working directory just before downloading them?
And slightly on the same topic, I'm stuck on how to verify the files exist before downloading them. I don't want to waste bandwidth or diskspace with empty files.. Here's what I have so far:
for (int x = 1; x <= 6000; x++)
{
pbsscount = x.ToString();
// Used for downloading file
string directoryName = textBox1.Text.ToString().Replace(":", "_");
if (!Directory.Exists(textBox1.Text))
Directory.CreateDirectory(directoryName.Substring(7));
string wholePBSSurl = textBox1.Text + "/" + "pb" + pbsscount.PadLeft(6, '0') + ".png";
// Used for saving file, file name in directory
string partPBSSurl = "pb" + pbsscount.PadLeft(6, '0') + ".png";
Uri uri2 = new Uri(wholePBSSurl);
//if (fileExists(wholePBSSurl))
//{
// Initialize downloading info, grab progressbar info
WebClient webClient = new WebClient();
webClient.DownloadFileCompleted += new AsyncCompletedEventHandler(Completed);
webClient.DownloadProgressChanged += new DownloadProgressChangedEventHandler(ProgressChanged);
// Save file to folder
//webClient.DownloadFileAsync(uri2, textBox1.Text + "/" + partPBSSurl);
webClient.DownloadFileAsync(uri2, partPBSSurl);
//}
}
Do I need to temporarily change the current working directory just before downloading them?
The second parameter can be a full path #"C:\folder\file.png". If you're fine with relative path to your current directory, just change the code to webClient.DownloadFileAsync(uri2, directoryName + partPBSSurl); or even better use System.Path.Combine(directoryName, partPBSSurl)
Sure you can know the size before If sever supports that. See: How to get the file size from http headers
I don't want to waste bandwidth or diskspace with empty files.
I wouldn't worry about that. The performance slow down is negligible.
There is no need to change the current directory. You are already using an overload of DownloadFileAsync that accepts a file path as the second parameter.
Just ensure that partPBSSurl contains a full path to the destination file, including both the directory and filename.
With regard to your second question of avoiding wasted time if the file does not exist, it so happens that I asked the same question recently:
Fail Fast with WebClient
Finally, I recently extended WebClient to provide simpler progress change events and allow for the timeout to be changed. I posed that code here:
https://stackoverflow.com/a/9763976/141172
When the user selects the list of files from a page and hit's download selected, then a post back happens to server and starts zipping on the server. This works great until we hit the timeout on the page ( which is default to 90 seconds ) and just returns the process to the page even though the backend process is still zipping. Is it possible to show the size of zip file when the file is being zipped instead of waiting till the end to provide the download link?
You can use ActiveX components to do that:
var oas = new ActiveXObject("Scripting.FileSystemObject");
var d = filepath;
var e = oas.getFile(d);
var f = e.size;
alert(f + " bytes");
}
but it will limit you to IE and should have appropriate IE security settings.