We Keep Coding

Incomplete/Corrupted downloaded files

I have made this little tool wich goes through a list of image links and download them to the hard drive, however, some of the pictures are incomplete (Check this picture) and they don't even raise an exception. The code below shows the download method I'm using in my tool.
private void Download(string url)
{
try
{
HttpWebRequest Request = WebRequest.Create(url) as HttpWebRequest;
Request.Method = WebRequestMethods.Http.Get;
Request.Timeout = 60 * 1000;
FileInfo ImageFile = new FileInfo(Path.Combine(BaseDirectory, Path.GetFileName(url)));
if (!ImageFile.Exists)
{
using (HttpWebResponse Response = Request.GetResponse() as HttpWebResponse)
{
if (Response.StatusCode.Equals(HttpStatusCode.OK))
{
using (FileStream FStream = new FileStream(ImageFile.FullName, FileMode.Create, FileAccess.Write, FileShare.None, 4096))
Response.GetResponseStream().CopyTo(FStream, 4096);
}
}
}
}
catch (Exception e)
{
Console.WriteLine("Error Downloading: {0}\r\nMessage: {1}", url, e.Message);
}
}
I can't figure out whether the problem is on the server side or there is something wrong with my code, so what do you think?

Have you tried calling Flush() after you've read the data? It looks like the last part of the stream isn't being written out.

Get a list from FTP folder on web (port 80) to array

I'm trying to get a list of files from FTP web folder via port 80 into an array or list and then download specific extensions, but a counter is always zero for some reason.
public void GetFilesFromServer(string url, string extension=".mp3")
{
List<string> files = new List<string>();
try
{
//Create FTP request
FtpWebRequest request = (FtpWebRequest)FtpWebRequest.Create(url);
request.Method = WebRequestMethods.Ftp.ListDirectory;
request.Credentials = new NetworkCredential("anonymous", "anonymous");
request.UsePassive = true;
request.UseBinary = true;
request.KeepAlive = false;
FtpWebResponse response = (FtpWebResponse)request.GetResponse();
Stream responseStream = response.GetResponseStream();
StreamReader reader = new StreamReader(responseStream);
while (!reader.EndOfStream)
{
files.Add(reader.ReadLine());
}
//Clean-up
reader.Close();
responseStream.Close(); //redundant
response.Close();
}
catch (Exception)
{
// MessageBox.Show("There was an error connecting to the FTP Server");
}
//If the list was successfully received, display it to the user
//through a dialog
if (files.Count != 0)
{
string folderPath = Path.Combine(AppDomain.CurrentDomain.BaseDirectory, "DownloadedFiles");
WebClient wc = new WebClient();
foreach (string file in files)
{
if(file.EndsWith(extension))
wc.DownloadFile(url+"/"+file, Path.Combine(folderPath, file));
}
}
}
My goal is to put all .ext files into an array and I can't get the list
The folder is http://url.com/folder for example.
But it fails to request
I checked the solutions and it doesn't work for me.

The FtpWebRequest is for FTP protocol.
FTP protocol does not use port 80.
If your URL is an HTML presentation of a folder on the server, you cannot scrape it with FtpWebRequest. You have to parse the HTML.
Though you should better find a way to access the data using a more reliable method (such as using a real FTP).

The MS documentation says that the FtpWebRequest initializes a new WebRequest and according to the documentation here : https://msdn.microsoft.com/en-us/library/bw00b1dc(v=vs.110).aspx
then if the URL starts with "http://" you will get back an HttpWebRequest rather than a FtpWebRequest.
This answer may explain:
Getting directory listing over http

Upload file with webrequest

Im trying to upload an image to an imagehost http://uploads.im/.
According to its very short API http://uploads.im/apidocs this is a way of doing that:
http://uploads.im/api?upload=http://www.google.com/images/srpr/nav_logo66.png
Note that in this example he is upploading an image from the internet and Im trying to upload a file from my computer.
Code:
public ActionResult SaveUploadedFile()
{
//Converts the image i want to upload to a bytearray
Image postData = img;
byte[] byteArray = imageToByteArray(postData);
//Is this adress not correct maybe? Is there a way to test?
WebRequest wrq = WebRequest.Create("http://uploads.im/api?upload=");
wrq.Method = ("POST");
//Im thinking that here I need som code
//that specifys the file i want to upload (bytearray)
using (WebResponse wrs = wrq.GetResponse())
using (Stream stream = wrs.GetResponseStream())
using (StreamReader reader = new StreamReader(stream))
{
string json = reader.ReadToEnd();
tempJson = json;
}
}
Please have look! Thanks!
EDIT, new code:
string filepath = #"c:\Users\xxxx\Desktop\Bilder\images\blank.gif";
using (WebClient client = new WebClient())
{
client.UploadFile("http://uploads.im/api?upload", filepath);
}
I get the error: : The underlying connection closed
EDIT with try catch:
string filepath = #"c:\Users\xxxx\Desktop\sack.png";
using (WebClient client = new WebClient())
{
try
{
client.UploadFile("http://uploads.im/api?upload", filepath);
}
catch (Exception e)
{
throw new ApplicationException(e);
}
}

private void UploadImage(string filepath)
{
using(WebClient uploader = new WebClient())
{
try
{
uploader.UploadFile(new Uri("http://uploads.im/api?upload"), filepath);
}
catch(Exception ex)
{
MessageBox.Show("An error occured :(\r\n" + ex.Message, "Error", MessageBoxButtons.OK, MessageBoxIcon.Error);
}
}
}
I'm using http://uploads.im/api?upload as the Endpoint, because, as read in the docs, it will be treated as REQUEST and will automatically detect the upload of an image.
However, I tried this code myself and get disconnected every-time without any meaningful response, just The remote host unexpectedly closed the connection. According to the docs you should get a Bad request when something is wrong. I contacted the support and hope they can tell me more about it. (Edit: They did not)

How can I combine my FTP queries?

My program has several times where it queries an FTP server to read and upload information. How can I combine all these into one FTP class to handle everything?
private static void UploadToFTP(string[] FTPSettings)
{
try
{
FtpWebRequest request = (FtpWebRequest)WebRequest.Create(FTPSettings[0]);
request.Method = WebRequestMethods.Ftp.MakeDirectory;
request.Credentials = new NetworkCredential(FTPSettings[1], FTPSettings[2]);
FtpWebResponse response = (FtpWebResponse)request.GetResponse();
}
catch
{
}
try
{
FtpWebRequest request = (FtpWebRequest)WebRequest.Create(FTPSettings[0] + Path.GetFileName(file));
request.Method = WebRequestMethods.Ftp.UploadFile;
request.Credentials = new NetworkCredential(FTPSettings[1], FTPSettings[2]);
StreamReader source = new StreamReader(file);
byte[] fileContents = Encoding.UTF8.GetBytes(source.ReadToEnd());
source.Close();
request.ContentLength = fileContents.Length;
Stream requestStream = request.GetRequestStream();
requestStream.Write(fileContents, 0, fileContents.Length);
requestStream.Close();
FtpWebResponse response = (FtpWebResponse)request.GetResponse();
response.Close();
RegenLog();
}
catch (Exception e)
{
File.AppendAllText(file, string.Format("{0}{0}Upload Failed - ({2}) - {1}{0}", nl, System.DateTime.Now, e.Message.ToString()));
}
}
private static void CheckBlacklist(string[] FTPSettings)
{
try
{
FtpWebRequest request = (FtpWebRequest)WebRequest.Create(FTPSettings[0] + "blacklist.txt");
request.Credentials = new NetworkCredential(FTPSettings[1], FTPSettings[2]);
using (WebResponse response = request.GetResponse())
{
using (Stream stream = response.GetResponseStream())
{
using (TextReader reader = new StreamReader(stream))
{
string blacklist = reader.ReadToEnd();
if (blacklist.Contains(Environment.UserName))
{
File.AppendAllText(file, string.Format("{0}{0}Logger terminated - ({2}) - {1}{0}", nl, System.DateTime.Now, "Blacklisted"));
uninstall = true;
}
}
}
}
}
catch (Exception e)
{
File.AppendAllText(file, string.Format("{0}{0}FTP Error - ({2}) - {1}{0}", nl, System.DateTime.Now, e.Message.ToString()));
}
}
private static void CheckUpdate(string[] FTPSettings)
{
try
{
FtpWebRequest request = (FtpWebRequest)WebRequest.Create(FTPSettings[0] + "update.txt");
request.Credentials = new NetworkCredential(FTPSettings[1], FTPSettings[2]);
using (WebResponse response = request.GetResponse())
{
using (Stream stream = response.GetResponseStream())
{
using (TextReader reader = new StreamReader(stream))
{
string newVersion = reader.ReadToEnd();
if (newVersion != version)
{
update = true;
}
}
}
}
}
catch (Exception e)
{
File.AppendAllText(file, string.Format("{0}{0}FTP Error - ({2}) - {1}{0}", nl, System.DateTime.Now, e.Message.ToString()));
}
}
I know my code is also a bit inconsistent and messy, however this is my first time working with FTP in C#. Please give any advice you have!

An ideal option would be to use a "manager" type of class, implementing the IDisposable interface.
Let it have some internal fields (which get initialized in the constructor) and one connection (also initialized in the constructor)
Every method should check if the connection is alive and, if it is, perform the operation; otherwise, try to restore it then throw an exception.
In Dispose method, you close the connection.
This should give you an idea on how it may look:
// Partial class declaration
class FTPManager : IDisposable
{
void Dispose();
FTPManager(string host, string username, string password)
void UploadFile(string remoteDir, string localfile);
void CreateDirectory(string remotePath);
void DeleteFile(string remotePath);
}
// and here I use it:
void Main()
{
using (var manager = new FTPManager("ftp.somehosting.org","login","password")) {
manager.CreateDirectory("/newfolder/");
manager.UploadFile("/newfolder/","C:\\Somefile.txt");
manager.DeleteFile("/newfolder/anotherfile.txt");
}
}

You might want to look at the WinSCP library.
WinSCP has a lot of features and supports more secure protocols than (inherently unsecure FTP).
Recent releases even feature a scripting mechanism.

You should use other FTP .NET library (there is a number of them over the internet, free ones and commercial as well), which allow to do all the commands in one connection. That would work much faster.

How to check if file exists on FTP before FtpWebRequest

I need to use FtpWebRequest to put a file in a FTP directory. Before the upload, I would first like to know if this file exists.
What method or property should I use to check if this file exists?

var request = (FtpWebRequest)WebRequest.Create
("ftp://ftp.domain.com/doesntexist.txt");
request.Credentials = new NetworkCredential("user", "pass");
request.Method = WebRequestMethods.Ftp.GetFileSize;
try
{
FtpWebResponse response = (FtpWebResponse)request.GetResponse();
}
catch (WebException ex)
{
FtpWebResponse response = (FtpWebResponse)ex.Response;
if (response.StatusCode ==
FtpStatusCode.ActionNotTakenFileUnavailable)
{
//Does not exist
}
}
As a general rule it's a bad idea to use Exceptions for functionality in your code like this, however in this instance I believe it's a win for pragmatism. Calling list on the directory has the potential to be FAR more inefficient than using exceptions in this way.
If you're not, just be aware it's not good practice!
EDIT: "It works for me!"
This appears to work on most ftp servers but not all. Some servers require sending "TYPE I" before the SIZE command will work. One would have thought that the problem should be solved as follows:
request.UseBinary = true;
Unfortunately it is a by design limitation (big fat bug!) that unless FtpWebRequest is either downloading or uploading a file it will NOT send "TYPE I". See discussion and Microsoft response here.
I'd recommend using the following WebRequestMethod instead, this works for me on all servers I tested, even ones which would not return a file size.
WebRequestMethods.Ftp.GetDateTimestamp

Because
request.Method = WebRequestMethods.Ftp.GetFileSize
may fails in some case (550: SIZE not allowed in ASCII mode), you can just check Timestamp instead.
reqFTP.Credentials = new NetworkCredential(inf.LogOn, inf.Password);
reqFTP.UseBinary = true;
reqFTP.Method = WebRequestMethods.Ftp.GetDateTimestamp;

FtpWebRequest (nor any other class in .NET) does not have any explicit method to check a file existence on FTP server. You need to abuse a request like GetFileSize or GetDateTimestamp.
string url = "ftp://ftp.example.com/remote/path/file.txt";
WebRequest request = WebRequest.Create(url);
request.Credentials = new NetworkCredential("username", "password");
request.Method = WebRequestMethods.Ftp.GetFileSize;
try
{
request.GetResponse();
Console.WriteLine("Exists");
}
catch (WebException e)
{
FtpWebResponse response = (FtpWebResponse)e.Response;
if (response.StatusCode == FtpStatusCode.ActionNotTakenFileUnavailable)
{
Console.WriteLine("Does not exist");
}
else
{
Console.WriteLine("Error: " + e.Message);
}
}
If you want a more straightforward code, use some 3rd party FTP library.
For example with WinSCP .NET assembly, you can use its Session.FileExists method:
SessionOptions sessionOptions = new SessionOptions {
Protocol = Protocol.Ftp,
HostName = "ftp.example.com",
UserName = "username",
Password = "password",
};
Session session = new Session();
session.Open(sessionOptions);
if (session.FileExists("/remote/path/file.txt"))
{
Console.WriteLine("Exists");
}
else
{
Console.WriteLine("Does not exist");
}
(I'm the author of WinSCP)

You can use WebRequestMethods.Ftp.ListDirectory to check if a file exist, no need for nasty try catch mechanism.
private static bool ExistFile(string remoteAddress)
{
int pos = remoteAddress.LastIndexOf('/');
string dirPath = remoteAddress.Substring(0, pos); // skip the filename only get the directory
NetworkCredential credentials = new NetworkCredential(FtpUser, FtpPass);
FtpWebRequest listRequest = (FtpWebRequest)WebRequest.Create(dirPath);
listRequest.Method = WebRequestMethods.Ftp.ListDirectory;
listRequest.Credentials = credentials;
using (FtpWebResponse listResponse = (FtpWebResponse)listRequest.GetResponse())
using (Stream listStream = listResponse.GetResponseStream())
using (StreamReader listReader = new StreamReader(listStream))
{
string fileToTest = Path.GetFileName(remoteAddress);
while (!listReader.EndOfStream)
{
string fileName = listReader.ReadLine();
fileName = Path.GetFileName(fileName);
if (fileToTest == fileName)
{
return true;
}
}
}
return false;
}
static void Main(string[] args)
{
bool existFile = ExistFile("ftp://123.456.789.12/test/config.json");
}

I use FTPStatusCode.FileActionOK to check if file exists...
then, in the "else" section, return false.