We Keep Coding

System.OutOfMemory exception when trying to read large files

public static byte[] ReadMemoryMappedFile(string fileName)
{
long length = new FileInfo(fileName).Length;
using (var stream = File.Open(fileName, FileMode.OpenOrCreate, FileAccess.Read, FileShare.ReadWrite))
{
using (var mmf = MemoryMappedFile.CreateFromFile(stream, null, length, MemoryMappedFileAccess.Read, null, HandleInheritability.Inheritable, false))
{
using (var viewStream = mmf.CreateViewStream(0, length, MemoryMappedFileAccess.Read))
{
using (BinaryReader binReader = new BinaryReader(viewStream))
{
var result = binReader.ReadBytes((int)length);
return result;
}
}
}
}
}
OpenFileDialog openfile = new OpenFileDialog();
openfile.Filter = "All Files (*.*)|*.*";
openfile.ShowDialog();
byte[] buff = ReadMemoryMappedFile(openfile.FileName);
texteditor.Text = BitConverter.ToString(buff).Replace("-"," "); <----A first chance exception of type 'System.OutOfMemoryException' occurred in mscorlib.dll
I get a System.OutOfMemory exception when trying to read large files.
I've read a lot for 4 weeks in all the web... and tried a lot!!! But still, I can't seem to find a good solution to my problem.
Please help me..
Update
public byte[] FileToByteArray(string fileName)
{
byte[] buff = null;
FileStream fs = new FileStream(fileName,
FileMode.Open,
FileAccess.Read);
BinaryReader br = new BinaryReader(fs);
long numBytes = new FileInfo(fileName).Length;
buff = br.ReadBytes((int)numBytes);
//return buff;
return File.ReadAllBytes(fileName);
}
OR
public static byte[] FileToByteArray(FileStream stream, int initialLength)
{
// If we've been passed an unhelpful initial length, just
// use 32K.
if (initialLength < 1)
{
initialLength = 32768;
}
BinaryReader br = new BinaryReader(stream);
byte[] buffer = new byte[initialLength];
int read = 0;
int chunk;
while ((chunk = br.Read(buffer, read, buffer.Length - read)) > 0)
{
read += chunk;
// If we've reached the end of our buffer, check to see if there's
// any more information
if (read == buffer.Length)
{
int nextByte = br.ReadByte();
// End of stream? If so, we're done
if (nextByte == -1)
{
return buffer;
}
// Nope. Resize the buffer, put in the byte we've just
// read, and continue
byte[] newBuffer = new byte[buffer.Length * 2];
Array.Copy(buffer, newBuffer, buffer.Length);
newBuffer[read] = (byte)nextByte;
buffer = newBuffer;
read++;
}
}
// Buffer is now too big. Shrink it.
byte[] ret = new byte[read];
Array.Copy(buffer, ret, read);
return ret;
}
I still get a System.OutOfMemory exception when trying to read large files.

If your file is 4GB, then BitConverter will turn each byte into XX- string, each char in string is 2 bytes * 3 chars per byte * 4 294 967 295 bytes = 25 769 803 770. You need +25Gb of free memory to fit entire string, plus you already have your file in memory as byte array.
Besides, no single object in a .Net program may be over 2GB. Theoretical limit for a string length would be 1,073,741,823 chars, but you also need to have a 64-bit process.
So solution in your case - open FileStream. Read first 16384 bytes (or how much can fit on your screen), convert to hex and display, and remember file offset. When user wants to navigate to next or previous page - seek to that position in file on disk, read and display again, etc.

You need to read the file in chunks, keep track of where you are in the file, page the contents on screen and use seek and position to move up and down in the file stream.

You will not be able to display 4Gb file reading all of it in memory first by any approach.
The approach is to virtualize the data, reading only the visible lines when user scrolls. If you need to do a read-only text viewer then you can use WPF ItemsControl with virtulizing stack panel and bind to custom IList collection which will lazily fetch lines from the file calculating file offset by for the line index.

Sending strings and File from Android Client to C# server

I am trying to send files with a bunch of strings from Android client to C# server. The strings, among other things, will also contain details with regards to the file being sent eg: File Size, File Name, etc. The issue I am having is that all the file bytes is not received thus the original file cannot be reconstructed at the destination even though I can properly retrieve all the strings.
My C# Server Code
while (true)
{
Socket socket = listener.AcceptSocket();
setStatus(socket.RemoteEndPoint + " Connected");
try
{
// Open the stream
NetworkStream stream = new NetworkStream(socket);
System.IO.StreamReader sr = new StreamReader(stream);
//Get string data
//********************************************
Teststr = sr.ReadLine();
setStatus(Teststr);
FileSize = sr.ReadLine();
long fileSizeLong = Convert.ToInt64(FileSize);
int length = (int)fileSizeLong;
setStatus("File size: " + length + " bytes");
//********************************************
//read bytes to buffer
byte[] buffer = new byte[length];
int toRead = (int)length;
int read = 0;
while (toRead > 0)
{
int noChars = stream.Read(buffer, read, toRead);
read += noChars;
toRead -= noChars;
}
setStatus("File Recieved. Total bytes: " + Convert.ToString(buffer.Length));
setStatus("Saving File");
String recievedPath = "C:\\Test\\";
BinaryWriter bWrite = new BinaryWriter(File.Open(recievedPath + "Test.png", FileMode.Create));
bWrite.Write(buffer);
setStatus("File Saved to: " + recievedPath);
bWrite.Flush();
bWrite.Close();
stream.Flush();
stream.Close();
}
catch (Exception e)
{
setStatus(e.Message);
}
setStatus("Disconnected");
socket.Close();
}
My Android Client Code
File file = new File(configstr[2]); //create file instance
try {
client = new Socket(configstr[0], Integer.valueOf(configstr[1]));
//Read file
fileInputStream = new FileInputStream(file);
outputStream = client.getOutputStream();
//Output database details to stream
//*****************************************
//Holds the string before conversion to bytes
String text = "";
text = "Test\n";
outputStream.write(text.getBytes());
text = String.valueOf(file.length()) + "\n";
outputStream.write(text.getBytes());
//*****************************************
outputStream.flush();
ByteArrayOutputStream bos = new ByteArrayOutputStream();
byte[] b = new byte[1024];
int bytesRead = 0;
while ((bytesRead = fileInputStream.read(b)) != -1) {
bos.write(b, 0, bytesRead);
}
byte[] bytes = bos.toByteArray();
outputStream.write(bytes);
outputStream.flush();
return true;
} catch (UnknownHostException e) {
e.printStackTrace();
return false;
} catch (IOException e) {
e.printStackTrace();
return false;
}
NOTE: The code works well if I only send one string, eg: If I only send file size.
Any better way of getting this to work as I am trying to send many strings through the stream as well as binary data from the file.

BufferedWriter with flush() at the end of every write() solved the issue for me.

FileResult buffered to memory

I'm trying to return large files via a controller ActionResult and have implemented a custom FileResult class like the following.
public class StreamedFileResult : FileResult
{
private string _FilePath;
public StreamedFileResult(string filePath, string contentType)
: base(contentType)
{
_FilePath = filePath;
}
protected override void WriteFile(System.Web.HttpResponseBase response)
{
using (FileStream fs = new FileStream(_FilePath, FileMode.Open, FileAccess.Read))
{
int bufferLength = 65536;
byte[] buffer = new byte[bufferLength];
int bytesRead = 0;
while (true)
{
bytesRead = fs.Read(buffer, 0, bufferLength);
if (bytesRead == 0)
{
break;
}
response.OutputStream.Write(buffer, 0, bytesRead);
}
}
}
}
However the problem I am having is that entire file appears to be buffered into memory. What would I need to do to prevent this?

You need to flush the response in order to prevent buffering. However if you keep on buffering without setting content-length, user will not see any progress. So in order for users to see proper progress, IIS buffers entire content, calculates content-length, applies compression and then sends the response. We have adopted following procedure to deliver files to client with high performance.
FileInfo path = new FileInfo(filePath);
// user will not see a progress if content-length is not specified
response.AddHeader("Content-Length", path.Length.ToString());
response.Flush();// do not add anymore headers after this...
byte[] buffer = new byte[ 4 * 1024 ]; // 4kb is a good for network chunk
using(FileStream fs = path.OpenRead()){
int count = 0;
while( (count = fs.Read(buffer,0,buffer.Length)) >0 ){
if(!response.IsClientConnected)
{
// network connection broke for some reason..
break;
}
response.OutputStream.Write(buffer,0,count);
response.Flush(); // this will prevent buffering...
}
}
You can change buffer size, but 4kb is ideal as lower level file system also reads buffer in chunks of 4kb.

Akash Kava is partly right and partly wrong. You DO NOT need to add the Content-Length header or do the flush afterward. But you DO, need to periodically flush response.OutputStream and then response. ASP.NET MVC (at least version 5) will automatically convert this into a "Transfer-Encoding: chunked" response.
byte[] buffer = new byte[ 4 * 1024 ]; // 4kb is a good for network chunk
using(FileStream fs = path.OpenRead()){
int count = 0;
while( (count = fs.Read(buffer,0,buffer.Length)) >0 ){
if(!response.IsClientConnected)
{
// network connection broke for some reason..
break;
}
response.OutputStream.Write(buffer,0,count);
response.OutputStream.Flush();
response.Flush(); // this will prevent buffering...
}
}
I tested it and it works.

Created Custom File Downloads. MD5 Hash Doesn't Match

Using Asp.Net MVC I was creating a file downloader. The problem with the built in Asp.Net MVC functions is that they don't work on extremely large file downloads and in some browsers they don't pop up the save-as dialog. So I rolled by own using an article from msdn http://support.microsoft.com/kb/812406. The problem now is that the files are downloading perfectly, but the MD5 Checksums aren't matching because the file size is slightly different on the server than the download (even though 1000 tests show that the downloads execute just fine). Here is the code:
public class CustomFileResult : ActionResult
{
public string File { get; set; }
public CustomFileResult(string file)
{
this.File = file;
}
public override void ExecuteResult(ControllerContext context)
{
Stream iStream = null;
// Buffer to read 10K bytes in chunk:
byte[] buffer = new Byte[10000];
// Length of the file:
int length;
// Total bytes to read:
long dataToRead;
// Identify the file name.
string filename = System.IO.Path.GetFileName(this.File);
try
{
// Open the file.
iStream = new System.IO.FileStream(this.File, System.IO.FileMode.Open,
System.IO.FileAccess.Read, System.IO.FileShare.Read);
// Total bytes to read:
dataToRead = iStream.Length;
context.HttpContext.Response.ContentType = "application/octet-stream";
context.HttpContext.Response.AddHeader("Content-Disposition", "attachment; filename=" + filename);
// Read the bytes.
while (dataToRead > 0)
{
// Verify that the client is connected.
if (context.HttpContext.Response.IsClientConnected)
{
// Read the data in buffer.
length = iStream.Read(buffer, 0, 10000);
// Write the data to the current output stream.
context.HttpContext.Response.OutputStream.Write(buffer, 0, length);
// Flush the data to the HTML output.
context.HttpContext.Response.Flush();
buffer = new Byte[10000];
dataToRead = dataToRead - length;
}
else
{
//prevent infinite loop if user disconnects
dataToRead = -1;
}
}
}
catch (Exception ex)
{
// Trap the error, if any.
context.HttpContext.Response.Write("Error : " + ex.Message);
}
finally
{
if (iStream != null)
{
//Close the file.
iStream.Close();
}
context.HttpContext.Response.Close();
}
}
}
And the execution:
return new CustomFileResult(file.FullName);

Try using the
Response.TransmitFile(string fileName)
method.
It's really good and has some things to avoid OutOfMemory expections as well.
http://msdn.microsoft.com/en-us/library/12s31dhy(v=vs.80).aspx

Turns out the issue was a missing header.
context.HttpContext.Response.AddHeader("Content-Length", iStream.Length.ToString());
Adding that header solved the problem.

Once starting to write to the OutputStream, try flushing the OutputStream itself instead of flushing the response:
context.HttpContext.Response.OutputStream.Flush()

Your problem is here:
length = iStream.Read(buffer, 0, 10000);
// Write the data to the current output stream.
context.HttpContext.Response.OutputStream.Write(buffer, 0, length);
Every loop you read into a buffer of exactly 10,000 bytes and write that to stream. That means every file that someone downloads will be in multiples of 10,000. So if I was to download a file that is 9,998 bytes from your site, the file I got would be 10,000 bytes. Meaning that the hash would never match. My file would have 2 null bytes at the end of it.
You need to add a check to make sure that the amount of data to read is >=10k, and if it is not, resize your byte to the exact amount that is left, and transmit that. that should fix the hash mismatch
try something like this:
if (context.HttpContext.Response.IsClientConnected)
{
// Read the data in buffer.
if (dataToRead>=10000)
{
byte[] buffer = new byte[10000];
length = 10000
context.HttpContext.Response.OutputStream.Write(buffer, 0, length);
}
else
{
byte[] buffer = new byte[dataToRead];
length = buffer.Length;
context.HttpContext.Response.OutputStream.Write(buffer, 0, length);
}
// Flush the data to the HTML output.
context.HttpContext.Response.Flush();
dataToRead = dataToRead - length;
}

how to split a wmv file

im doing a application in which i split a wmv file and transfer it to otherlocation(in 'x' kbs) .after the transfer gets completed the file doesnt play,it gives a message as the format is not supported.is there anyother way to do it.
sory i will explain what im doing now
i wrote an remote application,i want to transfer a .wmv file from one machine to other,i want to split the .wmv and send it to the remote machine and use it there.if i try to send the complete file means it will take lot of memory that seems very bad.so i want to split it and send it.but the file doesnt gets played it raises an exception the format is not supported.
the following is the code im doing i just done it in the local machine itself(not remoting):
try
{
FileStream fswrite = new FileStream("D:\\Movie.wmv", FileMode.Create);
int pointer = 1;
int bufferlength = 12488;
int RemainingLen = 0;
int AppLen = 0;
FileStream fst = new FileStream("E:\\Movie.wmv", FileMode.Open);
int TotalLen = (int)fst.Length;
fst.Close();
while (pointer != 0)
{
byte[] svid = new byte[bufferlength];
using (FileStream fst1 = new FileStream("E:\\Movie.wmv", FileMode.Open))
{
pointer = fst1.Read(svid, AppLen, bufferlength);
fst1.Close();
}
fswrite.Write(svid, 0, pointer);
AppLen += bufferlength;
RemainingLen = TotalLen-AppLen;
if(RemainingLen < bufferlength)
{
byte[] svid1 = new byte[RemainingLen];
using (FileStream fst2 = new FileStream("E:\\Movie.wmv", FileMode.Open))
{
pointer = fst2.Read(svid1, 0, RemainingLen);
fst2.Close();
}
fswrite.Write(svid, 0, pointer);
break;
}
}
fswrite.Close();
}
catch(Exception ex)
{
MessageBox.Show(ex.Message);
}

You'll probably find Good way to send a large file over a network in C# helpful.

Im going to make the assumtion your spliting the file when your sending it, and not trying to have the wmv in 3 different files on the remote machine.
When your sending the file what you basicly do is this:
Local machine
1) Read 16k bytes ( Or whatever number you prefere )
2) Send those 16k bytes over the network
3) Repeat above steps untill done
Remote machine
1) Listen for a connection
2) Get 16k bytes
3) Write 16k bytes
4) Repeat untill done.
This method will work, but your kind of inventing the wheel again, i would recommend using either something as simple as File.Copy ( Works fine over the network ) or if that does not meet your needs perhaps using a FTP client / server solution ( Plenty of C# examples on the net that can be hosted inside your application ).

i tried this
private void Splitinthread()
{
int bufferlength = 2048;
int pointer = 1;
int offset = 0;
int length = 0;
byte[] buff = new byte[2048];
FileStream fstwrite = new FileStream("D:\\TEST.wmv", FileMode.Create);
FileStream fst2 = new FileStream("E:\\karthi.wmv", FileMode.Open);
int Tot_Len = (int)fst2.Length;
int Remain_Buff = 0;
//Stream fst = File.OpenRead("E:\\karth.wmv");
while (pointer != 0)
{
try
{
fst2.Read(buff, 0, bufferlength);
fstwrite.Write(buff, 0, bufferlength);
offset += bufferlength;
Remain_Buff = Tot_Len - offset;
Fileprogress.Value = CalculateProgress(offset, Tot_Len);
if (Remain_Buff < bufferlength)
{
byte[] buff1 = new byte[Remain_Buff];
pointer = fst2.Read(buff1, 0, Remain_Buff);
fstwrite.Write(buff1, 0, pointer);
Fileprogress.Value = CalculateProgress(offset, Tot_Len);
fstwrite.Close();
fst2.Close();
break;
}
}
catch (Exception ex)
{
MessageBox.Show(ex.Message);
}
}
MessageBox.Show("Completed");
}