I have got a wcf service that sends me streams (large ones usually). As the client application my role is to get a stream over WCF and save it to disk. I've written some code but it seems like first getting the stream into ram and then write it to disk from ram. I want to safely get the stream and writing it directly to disk while not filling the ram with huge files. What is the good way of doing this? Here is what I did until now:
Stream sourceStream = SsClient.GetFile(FolderId, Helper.GetISession());
using (var targetStream = new FileStream(thisComputerPath, FileMode.OpenOrCreate, FileAccess.Write, FileShare.None))
{
//read from the input stream in 65000 byte chunks
const int bufferLen = 65000;
var buffer = new byte[bufferLen];
int count;
while ((count = sourceStream.Read(buffer, 0, bufferLen)) > 0)
{
// save to output stream
targetStream.Write(buffer, 0, count);
}
targetStream.Close();
sourceStream.Close();
}
I hope I could explain my problem clear enough. Excuse me for my english by the way.
I don't mind using ram for buffering purposes or something like that, i just don't want it to be filled with 1-2 gb of streams each time as it would give clients computer hard times if it just has 2 gb of ram.
Did you check the following posts
How to Save a Stream
and
Writing large stream to a file
Let us know incase of any queries on these implementations.
Related
I've written a program that was initially intended for very basic text communication over the internet using the .net TCPClient class in C#. I decided to try setting up a procedure to read a file from one computer, break it up into smaller pieces which are each sent to the receiving computer, and have it reassembled and saved there. Essentially a file transfer.
I then realized that all the data I'm transferring is going into the memory of the receiving computer and then onto the storage in the next step. I am now wondering, is this the best way to do it? If data can be transferred and immediately written to the storage location where it's headed (bypassing the RAM step), is this the way a program like Google Chrome would handle downloads? Or are there usually important reasons for the data to be stored in memory first?
By the way, for clarity, let's all agree that "storage" would be like a hard drive and "memory" refers to RAM. Thanks.
Th way it is done usually is you open a FileStream read data in byte[] from TcpClient and write the number of bytes read from NetworkStream to FileStream.
Here is a pseduso example :
TcpClient tcp;
FileStream fileStream = File.Open("WHERE_TO_SAVE", FileMode.Open, FileAccess.Write);
NetworkStream tcpStream = tcp.GetStream();
byte[] buffer = new byte[8192];
int bytesRead;
while ((bytesRead = tcpStream.Read(buffer, 0, buffer.Length)) > 0)
{
fileStream.Write(buffer, 0, bytesRead);
}
tcpStream.Dispose();
fileStream.Dispose();
I have this c# (.NET 4.0) application which uploads local files from the file system using a multi part form request. Here is the request stream:
Stream stream = webrequest.GetRequestStream();
and here is how I upload the file using the stream:
byte[] headerbytes = Encoding.UTF8.GetBytes(header);
stream.Write(headerbytes, 0, headerbytes.Length);
using (FileStream fileStream = new FileStream(file.FullName, FileMode.Open, FileAccess.Read))
{
byte[] buffer = new byte[1024];
int bytesRead = 0;
while ((bytesRead = fileStream.Read(buffer, 0, buffer.Length)) != 0)
{ stream.Write(buffer, 0, bytesRead); }
}
byte[] newlineBytes = Encoding.UTF8.GetBytes("\r\n");
stream.Write(newlineBytes, 0, newlineBytes.Length);
(the 'stream' is closed later on in the code')
My app runs and it hovers right around 20MB of RAM on the process. When I upload a 100MB file my app jumps to 120MB of RAM. When I upload a 700MB file my app jumps to 720MB of RAM. It does not seem to get garbage collected either. I profiled the app and stream.Write() is what is causing the excess memory allocation.
Is this a leak? What can I do?
The stream returned by the method GetRequestStream seems to be write-buffered by default. Together with relatively faster reading from the disk and slower writing to the network interface gives the effect of storing almost complete file in the memory.
You can either try to force the request stream to write the data to the underlying network stream by calling the method Flush or try to set the property AllowWriteStreamBuffering of the WebRequest to false.
I want to read a file into to stream as a base64string and I am using the below code to read the file , but when i see the memory usage is shooting over 900MB, when I monitor it through the task Manager. am I missing something, I could see that the memory usage shoots up when I read it into the string ,(I have used a 150 MB file to test the same).
stringBuilder Sb= new stringBuilder ();
using (var fs = new FileStream(#"C:\Users\Sanath Kumar\Desktop\s.pdf", FileMode.Open, FileAccess.Read))
{
var buffer = new byte[fs.Length];
fs.Read(buffer, 0, (int)fs.Length);
Sb.Append(Convert.ToBase64String(buffer));
buffer = null;
fs.Flush();
fs.Dispose();
}
The problem here is you are reading the file in it's entirety into memory and duplicating the data several times.
The byte[] array
The StringBuilder which has a char[] backing array
If this file is of a significant size this will definitely cause a spike in memory.
Instead of reading it all in at once try using the ToBase64Transform class instead. This is designed to work with a Stream as input and will avoid loading the entire file into memory at one time. Instead it will process it in chunks.
Please show me the best/fast methods for:
1) Loading very small binary files into memory. For example icons;
2) Loading/reading very big binary files of size 512Mb+.
3) Your common choice when you do not want to think about size/speed but must do only thing: read all bytes into memory?
Thank you!!!
P.S. Sorry for maybe trivial question. Please do not close it;)
P.S.2. Mirror of analog question for Java;
1: For very small files File.ReadAllBytes will be fine.
2: For very big files and using .net 4.0 , you can make use MemoryMapped Files.
3: If Not using .net 4.0 than , reading chunks of data would be good choice
1) I'd use a resource file rather than storing it as lots of separate files.
2) you probably want to stream the data rather than read it all at once, in which case you can use a FileStream.
3): Use ReadAllBytes:
byte[] bytes = File.ReadAllBytes(path);
1: For small, File.ReadAllBytes
2: For big, Stream (FileStream) or a BinaryReader on a Stream - the purpose being to remove the need to allocate a massive buffer, by changing the code to read small chunks consecutively
3: Go back and find the expected size; default to worst-case (#2)
Also note that I'd try to minimise the siE in the first place, perhaps via the choice of data-format, or compression.
This sample is good for both - for large files you need buffered reads.
public static byte[] ReadFile(string filePath)
{
byte[] buffer;
FileStream fileStream = new FileStream(filePath, FileMode.Open, FileAccess.Read);
try
{
int length = (int)fileStream.Length; // get file length
buffer = new byte[1024]; // create buffer
int count; // actual number of bytes read
int sum = 0; // total number of bytes read
// read until Read method returns 0 (end of the stream has been reached)
while ((count = fileStream.Read(buffer, sum, length - sum)) > 0)
sum += count; // sum is a buffer offset for next reading
}
finally
{
fileStream.Close();
}
return buffer;
}
I am trying to join a number of binary files that were split during download. The requirement stemmed from the project http://asproxy.sourceforge.net/. In this project author allows you to download files by providing a url.
The problem comes through where my server does not have enough memory to keep a file that is larger than 20 meg in memory.So to solve this problem i modified the code to not download files larger than 10 meg's , if the file is larger it would then allow the user to download the first 10 megs. The user must then continue the download and hopefully get the second 10 megs. Now i have got all this working , except when the user needs to join the files they downloaded i end up with corrupt files , as far as i can tell something is either being added or removed via the download.
I am currently join the files together by reading all the files then writing them to one file.This should work since i am reading and writing in bytes. The code i used to join the files is listed here http://www.geekpedia.com/tutorial201_Splitting-and-joining-files-using-C.html
I do not have the exact code with me atm , as soon as i am home i will post the exact code if anyone is willing to help out.
Please let me know if i am missing out anything or if there is a better way to do this , i.e what could i use as an alternative to a memory stream. The source code for the original project which i made changes to can be found here http://asproxy.sourceforge.net/download.html , it should be noted i am using version 5.0. The file i modified is called WebDataCore.cs and i modified line 606 to only too till 10 megs of data had been loaded the continue execution.
Let me know if there is anything i missed.
Thanks
You shouldn't split for memory reasons... the reason to split is usually to avoid having to re-download everything in case of failure. If memory is an issue, you are doing it wrong... you shouldn't be buffering in memory, for example.
The easiest way to download a file is simply:
using(WebClient client = new WebClient()) {
client.DownloadFile(remoteUrl, localPath);
}
Re your split/join code - again, the problem is that you are buffering everything in memory; File.ReadAllBytes is a bad thing unless you know you have small files. What you should have is something like:
byte[] buffer = new byte[8192]; // why not...
int read;
while((read = inStream.Read(buffer, 0, buffer.Length)) > 0)
{
outStream.Write(buffer, 0, read);
}
This uses a moderate buffer to pump data between the two as a stream. A lot more efficient. The loop says:
try to read some data (at most, the buffer-size)
(this will read at least 1 byte, or we have reached the end of the stream)
if we read something, write this many bytes from the buffer to the output
In the end i have found that by using a FTP request i was able to get arround the memory issue and the file is saved correctly.
Thanks for all the help
That example is loading each entire chunk into memory, instead you could do something like this:
int bufSize = 1024 * 32;
byte[] buffer = new byte[bufSize];
using (FileStream outputFile = new FileStream(OutputFileName, FileMode.OpenOrCreate,
FileAccess.Write, FileShare.None, bufSize))
{
foreach (string inputFileName in inputFiles)
{
using (FileStream inputFile = new FileStream(inputFileName, FileMode.Append,
FileAccess.Write, FileShare.None, buffer.Length))
{
int bytesRead = 0;
while ((bytesRead = inputFile.Read(buffer, 0, buffer.Length)) != 0)
{
outputFile.Write(buffer, 0, bytesRead);
}
}