I have to upload a large file to the server with the following code snippet:
static async Task LordNoBugAsync(string token, string filePath, string uri)
{
HttpWebRequest fileWebRequest = (HttpWebRequest)WebRequest.Create(uri);
fileWebRequest.Method = "PATCH";
fileWebRequest.AllowWriteStreamBuffering = false; //this line tells to upload by chunks
fileWebRequest.ContentType = "application/x-www-form-urlencoded";
fileWebRequest.Headers["Authorization"] = "PHOENIX-TOKEN " + token;
fileWebRequest.KeepAlive = false;
fileWebRequest.Timeout = System.Threading.Timeout.Infinite;
fileWebRequest.Proxy = null;
using (FileStream fileStream = File.OpenRead(filePath) )
{
fileWebRequest.ContentLength = fileStream.Length; //have to provide length in order to upload by chunks
int bufferSize = 512000;
byte[] buffer = new byte[bufferSize];
int lastBytesRead = 0;
int byteCount = 0;
Stream requestStream = fileWebRequest.GetRequestStream();
requestStream.WriteTimeout = System.Threading.Timeout.Infinite;
while ((lastBytesRead = fileStream.Read(buffer, 0, bufferSize)) != 0)
{
if (lastBytesRead > 0)
{
await requestStream.WriteAsync(buffer, 0, lastBytesRead);
//for some reasons didnt really write to stream, but in fact buffer has content, >60MB
byteCount += bufferSize;
}
}
requestStream.Flush();
try
{
requestStream.Close();
requestStream.Dispose();
}
catch
{
Console.Write("Error");
}
try
{
fileStream.Close();
fileStream.Dispose();
}
catch
{
Console.Write("Error");
}
}
...getting response parts...
}
In the code, I made a HttpWebRequest and push the content to server with buffering. The code works perfectly for any files under 60MB.
I tried a 70MB pdf. The buffer array has different content for each buffering. Yet, the request stream does not seem to be getting written. The bytecount also reached 70M, showing the file is properly read.
Edit (more info): I set the break point at requestStream.Close(). It clearly takes ~2 mins for the request stream to write in 60MB files but only takes 2ms for 70MB files.
My calling:
Task magic = LordNoBugAsync(token, nameofFile, path);
magic.Wait();
I am sure my calling is correct (it works for 0B to 60MB files).
Any advice or suggestion is much appreciated.
Related
I have an application which reads a file and copy's its content and writes into another file.
I am using buffer to read the file and write into another file.
The application take too long when the files are more.
Is there any specific optimal buffer size value that I can use to make the application more efficient?
I have used 256KB as maximum buffer size.
Below Upload method is called within a Parallel.ForEach loop.
Below is the code :
private bool Upload(string address, string uploadFile, string user, string password, string clientLogFile)
{
// Get the object used to communicate with the server.
FtpWebRequest request = null;
try
{
request = (FtpWebRequest)WebRequest.Create(address);
request.Credentials = new NetworkCredential(user, password);
request.Method = WebRequestMethods.Ftp.UploadFile;
request.KeepAlive = false;
request.Timeout = Convert.ToInt32(ConfigurationSettings.AppSettings["timeout"]);
request.UsePassive = Convert.ToBoolean(ConfigurationSettings.AppSettings["ftpMode"]);
// _fileBufferSize = 256kb
byte[] buffer = new byte[_fileBufferSize];
using (FileStream fs = new FileStream(uploadFile, FileMode.Open))
{
long dataLength = (long)fs.Length;
long bytesRead = 0;
int bytesDownloaded = 0;
using (Stream requestStream = request.GetRequestStream())
{
while (bytesRead < dataLength)
{
bytesDownloaded = fs.Read(buffer, 0, buffer.Length);
bytesRead = bytesRead + bytesDownloaded;
requestStream.Write(buffer, 0, bytesDownloaded);
}
requestStream.Close();
}
}
return true;
}
catch (Exception ex)
{
// Catch exception
}
finally
{
request = null;
}
return false;
}
All suggestions are welcome.
Consolidate your files to upload into multiple tasks to run, then execute several tasks in parallel.
Read this older guide from Microsoft on parallel tasks. A modern version of this might look like this code below.
public void UploadAllFiles(IEnumerable<FileUploadParameters> files) {
var tasks = new List<Task>();
foreach (var file in files) {
var task = Task.Run(() => {
UploadFile(file);
});
tasks.Add(task);
}
Task.WaitAll(tasks.ToArray());
}
I'm working on a project where I need to send large audio files via streams from a client to a server. I'm using the ASP.NET Web Api to communicate between client and server. My client has a "SendFile" method which I believe works fine, but I don't know how to make my server receive the data I'm sending via a stream. My client code looks like this so far:
private const int MAX_CHUNK_SIZE = (1024 * 5000);
private HttpWebRequest webRequest = null;
private FileStream fileReader = null;
private Stream requestStream = null;
public bool SendAudio(string uri, string file)
{
byte[] fileData;
fileReader = new FileStream(file, FileMode.Open, FileAccess.Read);
webRequest = (HttpWebRequest)WebRequest.Create(uri);
webRequest.Method = "POST";
webRequest.ContentLength = fileReader.Length;
webRequest.Timeout = 600000;
webRequest.Credentials = CredentialCache.DefaultCredentials;
webRequest.AllowWriteStreamBuffering = false;
requestStream = webRequest.GetRequestStream();
long fileSize = fileReader.Length;
long remainingBytes = fileSize;
int numberOfBytesRead = 0, done = 0;
while (numberOfBytesRead < fileSize)
{
SetByteArray(out fileData, remainingBytes);
done = WriteFileToStream(fileData, requestStream);
numberOfBytesRead += done;
remainingBytes -= done;
}
fileReader.Close();
return true;
}
public int WriteFileToStream(byte[] fileData, Stream requestStream)
{
int done = fileReader.Read(fileData, 0, fileData.Length);
requestStream.Write(fileData, 0, fileData.Length);
return done;
}
private void SetByteArray(out byte[] fileData, long bytesLeft)
{
fileData = bytesLeft < MAX_CHUNK_SIZE ? new byte[bytesLeft] : new byte[MAX_CHUNK_SIZE];
}
My server looks like this:
[HttpPost]
[ActionName("AddAudio")]
public async Task<IHttpActionResult> AddAudio([FromUri]string name)
{
try
{
isReceivingFile = true;
byte[] receivedBytes = await Request.Content.ReadAsByteArrayAsync();
if (WriteAudio(receivedBytes, name) == true)
{
isReceivingFile = false;
return Ok();
}
else
{
isReceivingFile = false;
return BadRequest("ERROR: Audio could not be saved on server.");
}
}
catch (Exception ex)
{
isReceivingFile = false;
return BadRequest("ERROR: Audio could not be saved on server.");
}
}
public bool WriteAudio(byte[] receivedBytes, string fileName)
{
string file = Path.Combine(#"C:\Users\username\Desktop\UploadedFiles", fileName);
using (FileStream fs = File.Create(file))
{
fs.Write(receivedBytes, 0, receivedBytes.Length);
}
return true;
}
The server code has the original code I wrote for it, before deciding to try and make it work with streams. The server code still works if I send a small file (under 30 MB), but if I send a large file my server gets a "outofmemoryexception". I can't figure out how to make the server take in the data via a stream. In my search for solutions I've come across a lot of examples with sockets and TCPClient, but that's not how we want to do it on this project. Can anybody help, please?
if I send a large file my server gets a "outofmemoryexception"
Well, it's reading the entire stream into memory right here:
byte[] receivedBytes = await Request.Content.ReadAsByteArrayAsync();
What you want to do is copy the stream from one location to another, without loading it all into memory at once. Something like this should work:
[HttpPost]
[ActionName("AddAudio")]
public async Task<IHttpActionResult> AddAudio([FromUri]string name)
{
try
{
string file = Path.Combine(#"C:\Users\username\Desktop\UploadedFiles", fileName);
using (FileStream fs = new FileStream(file, FileMode.Create, FileAccess.Write,
FileShare.None, 4096, useAsync: true))
{
await Request.Content.CopyToAsync(fs);
}
return Ok();
}
catch (Exception ex)
{
return BadRequest("ERROR: Audio could not be saved on server.");
}
}
I am writing an app in C# to measure and display download speed. I have the following code to download a 62MB file in chunks, which seems to work well for my purposes. I plan to extend this to measure the time required for each chunk, so it can be graphed.
Before doing so, I have a few questions to make sure this is actually doing what I think it is doing. Here is the code:
private void DownloadFile()
{
string uri = ConfigurationManager.AppSettings["DownloadFile"].ToString();
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(new Uri(uri));
int intChunkSize = 1048576; // 1 MB chunks
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
byte[] buffer = new byte[intChunkSize];
int intStatusCode = (int)response.StatusCode;
if (intStatusCode >= 200 && intStatusCode <= 299) // success
{
Stream sourceStream = response.GetResponseStream();
MemoryStream memStream = new MemoryStream();
int intBytesRead;
bool finished = false;
while (!finished)
{
intBytesRead= sourceStream.Read(buffer, 0, intChunkSize);
if (intBytesRead > 0)
{
memStream.Write(buffer, 0, intBytesRead);
// gather timing info here
}
else
{
finished = true;
}
}
}
}
}
The questions:
Does response contain all the data when it is instantiated, or just the header info? response.ContentLength does reflect the correct value.
Even though I am using a 1 MB chunk size, the actual bytes read (intBytesRead) in each iteration is much less, typically 16384 bytes (16 KB), but sometimes 1024 (1 KB). Why is this?
Is there any way to force it to actually read 1 MB chunks?
Does it serve any purpose here to actually write the data to the MemoryStream?
Thanks.
Dan
I want to download the first N Megabytes of huge XML File, so then I can close the broken tags with HTMLAgilityPack. Unfortunately, I can't use XMLReader.
I tried setting the Range on the HTTP Headers but that didn't seem to work, so now I'm trying this:
public string download(string url, int mb)
{
Int32 bytesToGet = 1048576 * mb;
HttpWebRequest request;
request = WebRequest.Create(url) as HttpWebRequest;
var buffer = new char[bytesToGet];
using (WebResponse response = request.GetResponse())
{
using (StreamReader sr = new StreamReader(response.GetResponseStream()))
{
sr.Read(buffer, 0, bytesToGet);
}
}
return new string(buffer);
}
but this still doesn't work either. I tried it with mb=5 and I get just a few lines of the XML file.
You're only calling Read() once, which doesn't promise to fill your buffer. Keep count of bytes downloaded and keep reading until your buffer is full or the end of the stream is reached:
int offset = 0;
int bytesRead = 0;
do
{
bytesRead = sr.Read(buffer, offset, bytesToGet - offset);
offset += bytesRead;
} while (bytesRead > 0);
I am developing c# application, in which i am downloading package(zip file) from server machine.It was downloading properly, but recently our package data has got some changes which is flex application.And by using c# we are downloading it into c drive or d drive.
Now with the new package i am facing some problem as
Unable to read data from the transport connection: An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full.
My code is below
byte[] packageData = null;
packageData = touchServerClient.DownloadFile("/packages/" + this.PackageName);
public byte[] DownloadFile(string url)
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(remoteSite.Url + url);
try
{
request.Method = "GET";
request.KeepAlive = false;
request.CookieContainer = new CookieContainer();
if (this.Cookies != null && this.Cookies.Count > 0)
request.CookieContainer.Add(this.Cookies);
HttpWebResponse webResponse = (HttpWebResponse)request.GetResponse();
// Console.WriteLine(response.StatusDescription);
Stream responseStream = webResponse.GetResponseStream();
int contentLength = Convert.ToInt32(webResponse.ContentLength);
byte[] fileData = StreamToByteArray(responseStream, contentLength);
return fileData;
}
public static byte[] StreamToByteArray(Stream stream, int initialLength)
{
// If we've been passed an unhelpful initial length, just
// use 32K.
if (initialLength < 1)
{
initialLength = 32768;
}
byte[] buffer = new byte[initialLength];
int read = 0;
int chunk;
while ((chunk = stream.Read(buffer, read, buffer.Length - read)) > 0)
{
read += chunk;
// If we've reached the end of our buffer, check to see if there's
// any more information
if (read == buffer.Length)
{
int nextByte = stream.ReadByte();
// End of stream? If so, we're done
if (nextByte == -1)
{
return buffer;
}
// Nope. Resize the buffer, put in the byte we've just
// read, and continue
byte[] newBuffer = new byte[buffer.Length * 2];
Array.Copy(buffer, newBuffer, buffer.Length);
newBuffer[read] = (byte)nextByte;
buffer = newBuffer;
read++;
}
}
// Buffer is now too big. Shrink it.
byte[] ret = new byte[read];
Array.Copy(buffer, ret, read);
return ret;
}
In the above function(StreamToByteArray) , i am getting error as
Unable to read data from the transport connection: An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full.
Please help me on this, coz i am not supposed to change the code also.
Thanks in advance
Sangita
A few things to try:
Wrap your stream handling in a using statement.
This will clean up that resource.
using (Stream responseStream = webResponse.GetResponseStream())
{
int contentLength = Convert.ToInt32(webResponse.ContentLength);
byte[] fileData = StreamToByteArray(responseStream, contentLength);
return fileData;
}
Make sure there are no other heavy memory processes running on the same box. Particularly if they are making Socket-bound calls.
Try upping the value of the MaxUserPort registry value. Here is the article if you didn't see the link provided in the comments.