We Keep Coding

Monitoring upload progress with HttpWebRequest

I've recently written a C# function that does a multi part form post for uploading files. To track the progress, I'd write the form data to the request stream at 4096 bytes at a time and call back with each write. However, it seems that the request does not even get sent until GetResponseAsync() is called.
If this is the case, is the reporting of every 4096 bytes written to the request stream an accurate reporting of upload progress?
If not, how can I accurately report progress? WebClient is out of the question for me, this is in a PCL Xamarin project.
private async Task<string> PostFormAsync (string postUrl, string contentType, byte[] formData)
{
try {
HttpWebRequest request = WebRequest.Create (postUrl) as HttpWebRequest;
request.Method = "POST";
request.ContentType = contentType;
request.Headers ["Cookie"] = Constants.Cookie;
byte[] buffer = new byte[4096];
int count = 0;
int length = 0;
using (Stream requestStream = await request.GetRequestStreamAsync ()) {
using (Stream inputStream = new MemoryStream (formData)) {
while ((count = await inputStream.ReadAsync (buffer, 0, buffer.Length)) > 0) {
await requestStream.WriteAsync (buffer, 0, count);
length += count;
Device.BeginInvokeOnMainThread (() => {
_progressBar.Progress = length / formData.Length;
});
}
}
}
_progressBar.Progress = 0;
WebResponse resp = await request.GetResponseAsync ();
using (Stream stream = resp.GetResponseStream ()) {
StreamReader respReader = new StreamReader (stream);
return respReader.ReadToEnd ();
}
} catch (Exception e) {
Debug.WriteLine (e.ToString ());
return String.Empty;
}
}
Please note that I am asking about monitoring progress of an upload at 4096 bytes at a time, not a download

I ended up accomplishing this by setting the AllowWriteStreamBuffering property of the WebRequest equal to false and the SendChunked property to true.
HOWEVER Xamarin.PCL (Profile 78) does not allow you to access these properties of the HttpWebRequest, so I had to instantiate my HttpWebRequest and return it from a dependency service in my platform specific project (only tested in iOS).
public class WebDependency : IWebDependency
{
public HttpWebRequest GetWebRequest(string uri)
{
var request = WebRequest.Create (uri) as HttpWebRequest;
request.SendChunked = true;
request.AllowWriteStreamBuffering = false;
return request;
}
}
And then to instantiate my web request -
HttpWebRequest request = DependencyService.Get<IWebDependency>().GetWebRequest(uri);

Progress in uploading in ftp server c#

I ve got the following code which
foreach (string dir in dirs) { //dirs all files in directory
try{
// Get an instance of WebClient
WebClient client = new System.Net.WebClient();
// parse the ftp host and file into a uri path for the upload
Uri uri = new Uri(m_FtpHost + new FileInfo(dir).Name);
// set the username and password for the FTP server
client.Credentials = new System.Net.NetworkCredential(m_FtpUsername, m_FtpPassword);
// upload the file asynchronously, non-blocking.
client.UploadFileAsync(uri, "STOR",dir);
}
catch(Exception e){
print(e.Message);
}
}
Can I retrieve back the progress of the upload? I have in the dirs 4-5 files. I want exact the progress (not the files uploaded/(total files))
EDIT: Thus the right approach is the following:
public int percentage;
try{
// Get an instance of WebClient
WebClient client = new System.Net.WebClient();
// parse the ftp host and file into a uri path for the upload
Uri uri = new Uri(m_FtpHost + new FileInfo(dir).Name);
// set the username and password for the FTP server
client.Credentials = new System.Net.NetworkCredential(m_FtpUsername, m_FtpPassword);
// upload the file asynchronously, non-blocking.
client.UploadProgressChanged += WebClientUploadProgressChanged;
client.UploadFileCompleted += WebClientUploadCompleted;
client.UploadFileAsync(uri, "STOR",dir);
}
void WebClientUploadProgressChanged(object sender, UploadProgressChangedEventArgs e)
{
percentage = e.ProgressPercentage;
}
void WebClientUploadCompleted(object sender, UploadFileCompletedEventArgs e)
{
print( "Upload is finished. ");
}
I add this implementation to my code, however it seems that it doenst print anything in the console.

WebClient contains a dedicated event for this
public event UploadProgressChangedEventHandler UploadProgressChanged
https://msdn.microsoft.com/en-us/library/system.net.webclient.uploadprogresschanged(v=vs.110).aspx
EDIT : HttpWebRequest approach based on a google result :
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uri);
request.Method = "POST";
request.ContentType = "text/plain";
request.Timeout = -1; //Infinite wait for the response.
// Get the file information object.
FileInfo fileInfo = new FileInfo("C:\\Test\\uploadFile.dat");
// Set the file content length.
request.ContentLength = fileInfo.Length;
// Get the number of segments the file will be uploaded to the stream.
int segments = Convert.ToInt32(fileInfo.Length / (1024 * 4));
// Get the source file stream.
using (FileStream fileStream = fileInfo.OpenRead())
{
// Create 4KB buffer which is file page size.
byte[] tempBuffer = new byte[1024 * 4];
int bytesRead = 0;
// Write the source data to the network stream.
using (Stream requestStream = request.GetRequestStream())
{
// Loop till the file content is read completely.
while ((bytesRead = fileStream.Read(tempBuffer, 0, tempBuffer.Length)) > 0)
{
// Write the 4 KB data in the buffer to the network stream.
requestStream.Write(tempBuffer, 0, bytesRead);
// Update your progress bar here using segment count.
}
}
}
// Post the request and Get the response from the server.
using (WebResponse response = request.GetResponse())
{
// Request is successfully posted to the server.
}

Low Download Speed when mouse is over Window

I have the following code to download some files from a FTP Server :
EDIT : I've solved the problem using DotNet, a good FTP WPF Library !
public partial class MainWindow
{
DispatcherTimer dispatcherTimer = new System.Windows.Threading.DispatcherTimer();
private byte[] downloadedData;
string FTPAddress = "ftp://ftp.cluster007.ovh.net";
double currentBytes;
double oldBytes;
public MainWindow()
{
InitializeComponent();
// DispatcherTimer setup
dispatcherTimer.Tick += new EventHandler(dispatcherTimer_Tick);
dispatcherTimer.Interval = new TimeSpan(0, 0, 1);
}
public static void DoEvents()
{
Application.Current.Dispatcher.Invoke(DispatcherPriority.Background,
new Action(delegate { }));
}
private void dispatcherTimer_Tick(object sender, EventArgs e)
{
currentBytes = Dl_ProgressBar.Value;
Dl_Speed.Content = "Vitesse : " + ((currentBytes - oldBytes) / 1000000).ToString("0.00") + " Mo/s";
oldBytes = Dl_ProgressBar.Value;
// Forcing the CommandManager to raise the RequerySuggested event
CommandManager.InvalidateRequerySuggested();
}
private void Dl_Button_Click(object sender, RoutedEventArgs e)
{
downloadFile();
}
private void downloadFile()
{
downloadedData = new byte[0];
try
{
//Create FTP request
//Note: format is ftp://server.com/file.ext
FtpWebRequest request = FtpWebRequest.Create(FTPAddress + "/" + filename) as FtpWebRequest;
//Get the file size first (for progress bar)
request.Method = WebRequestMethods.Ftp.GetFileSize;
request.Credentials = new NetworkCredential(username, password);
request.UsePassive = true;
request.UseBinary = true;
request.KeepAlive = true; //don't close the connection
int dataLength = (int)request.GetResponse().ContentLength;
Dl_Status.Content = "Téléchargement en cours...";
DoEvents();
//Now get the actual data
request = FtpWebRequest.Create(FTPAddress + "/" + filename) as FtpWebRequest;
request.Method = WebRequestMethods.Ftp.DownloadFile;
request.Credentials = new NetworkCredential(username, password);
request.UsePassive = true;
request.UseBinary = true;
request.KeepAlive = false; //close the connection when done
//Set up progress bar
Dl_ProgressBar.Value = 0;
Dl_ProgressBar.Maximum = dataLength;
//Streams
FtpWebResponse response = request.GetResponse() as FtpWebResponse;
Stream reader = response.GetResponseStream();
//Download to memory
//Note: adjust the streams here to download directly to the hard drive
MemoryStream memStream = new MemoryStream();
byte[] buffer = new byte[1024]; //downloads in chuncks
dispatcherTimer.Start();
while (true)
{
DoEvents(); //prevent application from crashing
int bytesRead = reader.Read(buffer, 0, buffer.Length);
if (bytesRead == 0)
{
//Nothing was read, finished downloading
Dl_ProgressBar.Value = Dl_ProgressBar.Maximum;
Dl_Percent.Content = "Progression : 100%";
DoEvents();
break;
}
else
{
//Write the downloaded data
memStream.Write(buffer, 0, bytesRead);
//Update the progress bar
if (Dl_ProgressBar.Value + bytesRead <= Dl_ProgressBar.Maximum)
{
Dl_ProgressBar.Value += bytesRead;
Dl_Percent.Content = "Progression : " + ((Dl_ProgressBar.Value / 1000000000000000) * dataLength).ToString("0.00") + "%";
DoEvents();
}
}
}
//Convert the downloaded stream to a byte array
downloadedData = memStream.ToArray();
//Clean up
reader.Close();
memStream.Close();
response.Close();
Dl_Status.Content = "Téléchargement terminé";
DoEvents();
}
catch (Exception)
{
Dl_Status.Content = "Erreur de connexion au FTP";
}
}
}
My problem is that when I pass the mouse over the window, the download speed is dropping significantly...
It changes from 3.70Mb/s to 2.20Mb/s.
When I have the mouse out of the window, there's no problem, but when I'm over it, it slows down, particularly when I do some very short movements, the download speed go to 0.20Mb/s.
I've tried to use Threads and Dispatcher but it was the same.

To answer your specific question, WPF's Dispatcher uses a priority queue, and Input level events (like those originating from mouse movement) take priority over Background level events. Your DoEvents() method periodically drains the message queue of all events with Background priority or higher, so when you move the mouse over the window, the queue fills up with input events to process. This means that DoEvents takes longer to return, and more time elapses before you can resume processing the download.
That said, this is a terrible way to accomplish a download; you should never use this kind of DoEvents() hack in WPF; do some research on the async and await features of C# (or, if that is not an option, BackgroundWorker). You will find many examples on StackOverflow of how to perform asynchronous downloads without having to resort to this sort of Dispatcher trickery to keep the UI responsive.

Executing web requests inside of a loop only works on the first pass

I am taking a string message and breaking it up into chunks so that I can send it to an sms service (that consequently doesn't break it up for you). I after I do my work for break those messages up, I try loop through the resulting array and execute a web request. The problem is that it only works for the first message, and then hangs after that. After a short time, I get an error message saying "The connection was closed unexpectedly." This occurs at the second time it attempts GetResponse(); I've seen a few other posts on here that were simply saying to close and dispose the response and request streams. This isn't working for me at all. This is where my code is currently:
private static void Main(string[] args)
{
var oldMessage = GetFileString();
Console.WriteLine(string.Format("Old message: {0}", oldMessage.Length));
var newMessage = UrlPathEncodeString(oldMessage);
Console.WriteLine(string.Format("New message: {0}", newMessage.Length));
var brokenUp = SplitByLength(newMessage, 145).ToArray();
for(var i = 0; i < brokenUp.Count(); i++)
{
brokenUp[i] = brokenUp[i].Insert(0, UrlPathEncodeString(string.Format("({0:D2} of {1:D2})", i + 1, brokenUp.Count())));
Console.WriteLine(string.Format("Appended length: {0}", brokenUp[i].Length));
}
System.Net.ServicePointManager.DefaultConnectionLimit = 100;
foreach (var block in brokenUp)
{
Thread.Sleep(1500);
SendSms((HttpWebRequest)WebRequest.Create("http://172.20.5.214:90/method/sendsms"), block);
}
Console.ReadKey();
}
public static void SendSms(HttpWebRequest request, string message)
{
//build the request
var url = "http://ipaddress/method/sendsms";
//var request = (HttpWebRequest)WebRequest.Create(url);
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded";
var fields = "CellNumber={0}&Message={1}";
fields = string.Format(fields, "16021234567", message);
var fieldsBytes = Encoding.UTF8.GetBytes(fields);
request.ContentLength = fieldsBytes.Length;
var length = fieldsBytes.Length;
using (var requestStream = request.GetRequestStream())
{
requestStream.Write(fieldsBytes, 0, length);
using (var response = request.GetResponse())
{
using (var responseStream = response.GetResponseStream())
{
responseStream.Close();
}
}
requestStream.Close();
}
}
public static byte[] ReadFully(Stream stream)
{
var buffer = new byte[32768];
using (var ms = new MemoryStream())
{
while (true)
{
int read = stream.Read(buffer, 0, buffer.Length);
if (read <= 0)
return ms.ToArray();
ms.Write(buffer, 0, read);
}
}
}

I've run into cases where response.Close appears to hang if you fail to download the entire contents of the response. Why this is, I don't know, but placing a call to request.Abort before calling Close() solves the problem. I wouldn't expect you to be seeing this problem, though, unless the response could potentially be many megabytes in size.
Also, failing to close the request stream before calling GetResponse might prevent all of the data from being sent. I would suggest calling requestStream.Close before making the request. But again, it seems odd that your code would work the first time but not on subsequent requests.
Your modified code, taking into account the changes I suggested, would be:
using (var requestStream = request.GetRequestStream())
{
requestStream.Write(fieldsBytes, 0, length);
}
using (var response = request.GetResponse())
{
using (var responseStream = response.GetResponseStream())
{
// read the response here.
request.Abort();
responseStream.Close();
}
}

Download file from FTP with Progress - TotalBytesToReceive is always -1?

I am trying to download a file from an FTP server with a progress bar.
The file is downloading, and the ProgressChanged event is calling, except in the event args TotalBytesToReceive is always -1. TotalBytes increases, but I am unable to calculate the percentage without the total.
I imagine I could find the file size through other ftp commands, but I wonder why this doesn't work?
My code:
FTPClient request = new FTPClient();
request.Credentials = credentials;
request.DownloadProgressChanged += new DownloadProgressChangedEventHandler(request_DownloadProgressChanged);
//request.DownloadDataCompleted += new DownloadDataCompletedEventHandler(request_DownloadDataCompleted);
request.DownloadDataAsync(new Uri(folder + file));
while (request.IsBusy) ;
....
static void request_DownloadProgressChanged(object sender, DownloadProgressChangedEventArgs e)
{
if (e.TotalBytesToReceive == -1)
{
l.reportProgress(-1, FormatBytes(e.BytesReceived) + " out of ?" );
}
else
{
l.reportProgress(e.ProgressPercentage, "Downloaded " + FormatBytes(e.BytesReceived) + " out of " + FormatBytes(e.TotalBytesToReceive) + " (" + e.ProgressPercentage + "%)");
}
}
....
class FTPClient : WebClient
{
protected override WebRequest GetWebRequest(System.Uri address)
{
FtpWebRequest req = (FtpWebRequest)base.GetWebRequest(address);
req.UsePassive = false;
return req;
}
}
Thanks.

So I had the same issue. I got around it by retrieving the file size first.
// Get the object used to communicate with the server.
FtpWebRequest request = (FtpWebRequest)WebRequest.Create("URL");
request.Method = WebRequestMethods.Ftp.GetFileSize;
request.Credentials = networkCredential;
FtpWebResponse response = (FtpWebResponse)request.GetResponse();
Stream responseStream = response.GetResponseStream();
bytes_total = response.ContentLength; //this is an int member variable stored for later
Console.WriteLine("Fetch Complete, ContentLength {0}", response.ContentLength);
response.Close();
webClient = new MyWebClient();
webClient.Credentials = networkCredential; ;
webClient.DownloadDataCompleted += new DownloadDataCompletedEventHandler(FTPDownloadCompleted);
webClient.DownloadProgressChanged += new DownloadProgressChangedEventHandler(FTPDownloadProgressChanged);
webClient.DownloadDataAsync(new Uri("URL"));
Then do the math in the callback.
private void FTPDownloadProgressChanged(object sender, DownloadProgressChangedEventArgs e)
{
progressBar.Value = (int)(((float)e.BytesReceived / (float)bytes_total) * 100.0);
}

With FTP protocol, WebClient in general does not know total download size. So you commonly get -1 with FTP.
Note that the behavior actually contradicts the .NET documentation, which says for FtpWebResponse.ContentLength (where the value of TotalBytesToReceive comes from):
For requests that use the DownloadFile method, the property is greater than zero if the downloaded file contained data and is zero if it was empty.
But you will easily find out many of questions about this, effectively showing that the behavior is not always as documented. The FtpWebResponse.ContentLength has a meaningful value for GetFileSize method only.
The FtpWebRequest/WebClient makes no explicit attempt to find out a size of the file that it is downloading. All it does is that it tries to look for (xxx bytes). string in 125/150 responses to RETR command. No FTP RFC mandates that the server should include such information. ProFTPD (see data_pasv_open in src/data.c) and vsftpd (see handle_retr in postlogin.c) seem to include this information. Other common FTP servers (IIS, FileZilla) do not do this.
If your server does not provide size information, you have to query for size yourself before download. A complete solution using FtpWebRequest and Task:
private void button1_Click(object sender, EventArgs e)
{
// Run Download on background thread
Task.Run(() => Download());
}
private void Download()
{
try
{
const string url = "ftp://ftp.example.com/remote/path/file.zip";
NetworkCredential credentials = new NetworkCredential("username", "password");
// Query size of the file to be downloaded
WebRequest sizeRequest = WebRequest.Create(url);
sizeRequest.Credentials = credentials;
sizeRequest.Method = WebRequestMethods.Ftp.GetFileSize;
int size = (int)sizeRequest.GetResponse().ContentLength;
progressBar1.Invoke(
(MethodInvoker)(() => progressBar1.Maximum = size));
// Download the file
WebRequest request = WebRequest.Create(url);
request.Credentials = credentials;
request.Method = WebRequestMethods.Ftp.DownloadFile;
using (Stream ftpStream = request.GetResponse().GetResponseStream())
using (Stream fileStream = File.Create(#"C:\local\path\file.zip"))
{
byte[] buffer = new byte[10240];
int read;
while ((read = ftpStream.Read(buffer, 0, buffer.Length)) > 0)
{
fileStream.Write(buffer, 0, read);
int position = (int)fileStream.Position;
progressBar1.Invoke(
(MethodInvoker)(() => progressBar1.Value = position));
}
}
}
catch (Exception e)
{
MessageBox.Show(e.Message);
}
}
The core download code is based on:
Upload and download a binary file to/from FTP server in C#/.NET

FTP Wont give you content sizes like HTTP does, you would probably be better doing this on your own.
FtpWebRequest FTPWbReq = WebRequest.Create("somefile") as FtpWebRequest;
FTPWbReq .Method = WebRequestMethods.Ftp.GetFileSize;
FtpWebResponse FTPWebRes = FTPWbReq.GetResponse() as FtpWebResponse;
long length = FTPWebRes.ContentLength;
FTPWebRes.Close();