We Keep Coding

Read a large binary file(5GB) into a byte array in C#?

I have a recording file (Binary file) more than 5 GB, i have to read that file and filter out the data needed to be send to server.
Problem is byte[] array supports till 2GB of file data . so just need help if someone had already dealt with this type of situation.
using (FileStream str = File.OpenRead(textBox2.Text))
{
int itemSectionStart = 0x00000000;
BinaryReader breader = new BinaryReader(str);
breader.BaseStream.Position = itemSectionStart;
int length = (int)breader.BaseStream.Length;
byte[] itemSection = breader.ReadBytes(length ); //first frame data
}
issues:
1: Length is crossing the range of integer.
2: tried using long and unint but byte[] only supports integer
Edit.
Another approach i want to give try, Read data on frame buffer basis, suppose my frame buffer size is 24000 . so byte array store that many frames data and then process the frame data and then flush out the byte array and store another 24000 frame data. till keep on going till end of binary file..

See you can not read that much big file at once, so you have to either split the file in small portions and then process the file.
OR
Read file using buffer concept and once you are done with that buffer data then flush out that buffer.
I faced the same issue, so i tried the buffer based approach and it worked for me.
FileStream inputTempFile = new FileStream(Path, FileMode.OpenOrCreate, FileAccess.Read);
Buffer_value = 1024;
byte[] Array_buffer = new byte[Buffer_value];
while ((bytesRead = inputTempFile.Read(Array_buffer, 0, Buffer_value)) > 0)
{
for (int z = 0; z < Array_buffer.Length; z = z + 4)
{
string temp_id = BitConverter.ToString(Array_buffer, z, 4);
string[] temp_strArrayID = temp_id.Split(new char[] { '-' });
string temp_ArraydataID = temp_strArrayID[0] + temp_strArrayID[1] + temp_strArrayID[2] + temp_strArrayID[3];
}
}
this way you can process your data.
For my case i was trying to store buffer read data in to a List, it will work fine till 2GB data after that it will throw memory exception.
The approach i followed, read the data from buffer and apply needed filters and write filter data in to a text file and then process that file.
//text file approach
FileStream inputTempFile = new FileStream(Path, FileMode.OpenOrCreate, FileAccess.Read);
Buffer_value = 1024;
StreamWriter writer = new StreamWriter(Path, true);
byte[] Array_buffer = new byte[Buffer_value];
while ((bytesRead = inputTempFile.Read(Array_buffer, 0, Buffer_value)) > 0)
{
for (int z = 0; z < Array_buffer.Length; z = z + 4)
{
string temp_id = BitConverter.ToString(Array_buffer, z, 4);
string[] temp_strArrayID = temp_id.Split(new char[] { '-' });
string temp_ArraydataID = temp_strArrayID[0] + temp_strArrayID[1] + temp_strArrayID[2] + temp_strArrayID[3];
if(temp_ArraydataID =="XYZ Condition")
{
writer.WriteLine(temp_ArraydataID);
}
}
}
writer.Close();

As said in comments, I think you have to read your file with a stream. Here is how you can do this:
int nbRead = 0;
var step = 10000;
byte[] buffer = new byte[step];
do
{
nbRead = breader.Read(buffer, 0, step);
hugeArray.Add(buffer);
foreach(var oneByte in hugeArray.SelectMany(part => part))
{
// Here you can read byte by byte this subpart
}
}
while (nbRead > 0);
If I well understand your needs, you are looking for a specific pattern into your file?
I think you can do it by looking for the start of your pattern byte by byte. Once you find it, you can start reading the important bytes. If the whole important data is greater than 2GB, as said in the comments, you will have to send it to your server in several parts.

how to get PCM data from stereo channel mp3 using Naudio in C#

I am new to Naudio and using it to get PCM data from Mp3 files, this is my code to take PCM from mono-channel file, but don't know how to do it with stereo channel file
code:
Mp3FileReader file = new Mp3FileReader(op.FileName);
int _Bytes = (int)file.Length;
byte[] Buffer = new byte[_Bytes];
file.Read(Buffer, 0, (int)_Bytes);
for (int i = 0; i < Buffer.Length - 2; i += 2)
{
byte[] Sample_Byte = new byte[2];
Sample_Byte[0] = Buffer[i + 1];
Sample_Byte[1] = Buffer[i + 2];
Int16 _ConvertedSample = BitConverter.ToInt16(Sample_Byte, 0);
}
How can I get PCM from stereo channel Mp3 file?

In a stereo file, the samples are interleaved: one left channel sample followed by one right channel etc. So in your loop you could go through four bytes at a time to read out the samples.
Also there are some bugs in your code. You should use return value of Read, not the size of the buffer, and you have an off by one error in the code to access the samples. Also, no need to copy into a temporary buffer.
Something like this should work for you:
var file = new Mp3FileReader(fileName);
int _Bytes = (int)file.Length;
byte[] Buffer = new byte[_Bytes];
int read = file.Read(Buffer, 0, (int)_Bytes);
for (int i = 0; i < read; i += 4)
{
Int16 leftSample = BitConverter.ToInt16(Buffer, i);
Int16 rightSample = BitConverter.ToInt16(Buffer, i + 2);
}

c# converting short array from mono wav file, to short array to write as stereo wav

This may be a really simple question; I converted a mono WAV file to a short[] array, and I have a function that writes this back to a WAV file. all works good. (writeBuffer is the short[] array)
byte[] dataBuffer = new byte[writeBuffer.Length * 2];
Buffer.BlockCopy(writeBuffer, 0, dataBuffer, 0, dataBuffer.Length);
using (var targetFile = isoStore.CreateFile("temp.wav"))
{
WriteHeader(targetFile, (int)dataBuffer.Length, 1, SamplesPerSecond);
int start = 0;
targetFile.Write(dataBuffer, start, (int)dataBuffer.Length);
targetFile.Flush();
targetFile.Close();
}
My question is what I need to do to that short array in order to have it written as a stereo file; If I write to my WAV header that the WAV has 2 channels (right now I am writing it has 1), by changing the following line:
WriteHeader(targetFile, (int)dataBuffer.Length, 2, SamplesPerSecond);
what do I need to do to my short[] array? will, for example, every second element be treated as part of the second channel? so would I need to create a short[] of writeBuffer.Length*2, and then write each value of writeBuffer to the new array as such:
short[] newBuffer = new short[writeBuffer.Length*2];
for (int i=0; i< newBuffer.Length;i = i + 2)
{
newBuffer[i] = writeBuffer[i];
newBuffer[i+1] = writeBuffer[i];
}
would this be correct? Or am I making an incorrect assumption about how the short[] array is to be written for a stereo file versus a mono file?

Change your code to:
short[] newBuffer = new short[writeBuffer.Length*2];
int index = offset; //offset might be 0
for (int i=0; i< newBuffer.Length;i = i + 2)
{
newBuffer[index++] = writeBuffer[i];
newBuffer[index++] = writeBuffer[i];
}

Array size based on available physical memory

I am trying to make an encryption algorithm.
I can read a file and convert it to bytes without any problems, and am saving the bytes in a byteArray.
The problem is I am currently creating the array size like this:
byte[] FileArray =new byte[10000000];
FileStream TheFileStream = new FileStream(FilePath.Text, FileMode.Open);
BinaryReader TheFileBinary = new BinaryReader(TheFileStream);
for (int i = 0; i < TheFileStream.Length; i++) {
FileArray = TheFileBinary.ReadBytes(10000000);
// I call a function here
if (TheFileStream.Position == TheFileStream.Length)
break;
}
However, I don't want the array size to be fixed, because if I make it 1000000 (as an example), other machines with small memory size might face a problem.
I need to find the Idle size of a memory size for each machine, how can I set the array size dynamically based on the free unallocated memory space, to be used where I can put it in the byteArray?
I have noticed the larger the Arraysize the faster it reads, so I don't want to make it too small either.
I would really appreciate the help.

The FileStream keeps track of how many bytes are in the file. Just use the Length property.
FileStream TheFileStream = new FileStream(FilePath.Text, FileMode.Open);
BinaryReader TheFileBinary = new BinaryReader(TheFileStream);
byte[] FileArray = TheFileBinary.ReadBytes(TheFileStream.Length);
Okay reread the question and finnaly found the part of it that was a question, "how can I know the free unallocated memory space so I can put it in the byteArray". Anyways I suggest you take a look at this question along with its highest rated comment.

If you're really worried about space, then use a simple List, read in chunks of the stream at a time (say 1024), and call the AddRange method on the list. After you're done, call ToArray on the List, and now you have a properly size byte array.
List<byte> byteArr = new List<byte>();
byte[] buffer = new byte[1024];
int bytesRead = 0;
using(FileStream TheFileStream = new FileStream(FilePath.Text, FileMode.Open))
{
while((bytesRead = stream.Read(buffer, 0, buffer.Length)) > 0)
byteArr.AddRange(buffer);
}
buffer = byteArr.ToArray();
// call your method here.
Edit: It's still preferable to read it in chunks for larger files. You can of course play with the buffer size however you want, but 1024 is usually a good starting point. Doing a read of the entire file will ultimately DOUBLE the memory, as you also have to deal with the internal read buffer being the size of the stream (on top of your own buffer). Breaking up the reads into chunks only takes FileStream.Length + <buffer size> memory as opposed to FileStream.Length * 2. Just something to keep in mind...
byte[] buffer = null;
using(FileStream TheFileStream = new FileStream(FilePath.Text, FileMode.Open))
{
buffer = new byte[TheFileStream.Length];
int offset = 0;
while((bytesRead = stream.Read(buffer, offset, 1024)) > 0)
offset += bytesRead;
// Or just TheFileStream.Read(buffer, 0, buffer.Length) if it's small enough.
}

You can use WMI to retrieve the instance of the Win32_OperatingSystem class and base your memory calculations off of the FreePhysicalMemory or TotalVisibleMemorySize properties:
static ulong GetAvailableMemoryKilobytes()
{
const string memoryPropertyName = "FreePhysicalMemory";
using (ManagementObject operatingSystem = new ManagementObject("Win32_OperatingSystem=#"))
return (ulong) operatingSystem[memoryPropertyName];
}
static ulong GetTotalMemoryKilobytes()
{
const string memoryPropertyName = "TotalVisibleMemorySize";
using (ManagementObject operatingSystem = new ManagementObject("Win32_OperatingSystem=#"))
return (ulong) operatingSystem[memoryPropertyName];
}
Then pass the result of either method to a method like this to scale the size of your read buffer to the memory of the local machine:
static int GetBufferSize(ulong memoryKilobytes)
{
const int bufferStepSize = 256; // 256 kilobytes of buffer...
const int memoryStepSize = 128 * 1024;// ...for every 128 megabytes of memory...
const int minBufferSize = 512; // ...no less than 512 kilobytes...
const int maxBufferSize = 10 * 1024; // ...no more than 10 megabytes
int bufferSize = bufferStepSize * ((int) memoryKilobytes / memoryStepSize);
bufferSize = Math.Max(bufferSize, minBufferSize);
bufferSize = Math.Min(bufferSize, maxBufferSize);
return bufferSize;
}
Obviously, increasing your buffer size by 256 KB for every 128 MB of RAM seems a little silly, but these number are just examples of how you might scale your buffer size if you really wanted to do that. Unless you're reading many, many files at once, worrying about a buffer that's a few hundred kilobytes or a few megabytes might be more trouble than it's worth. You might be better off just benchmarking to see which sized buffer gives the best performance (it might not need to be as large as you think) and using that.
Now you can simply update your code like this:
ulong memoryKilobytes =
GetAvailableMemoryKilobytes();
// ...or GetTotalMemoryKilobytes();
int bufferSize = GetBufferSize(memoryKilobytes);
using (FileStream TheFileStream = new FileStream(FilePath.Text, FileMode.Open))
{
byte[] FileArray = new byte[bufferSize];
int readCount;
while ((readCount = TheFileBinary.Read(FileArray, 0, bufferSize)) > 0)
{
// Call a method here, passing FileArray as a parameter
}
}

C# equivalent of fread

I am in the process of converting some C++ code to C#, I am trying to figure out how I could write out and following C++ code in my C# app and have it do the same thing:
fread(&Start, 1, 4, ReadMunge); //Read File position
I have tried multiple ways such as using FileStream:
using (FileStream fs = File.OpenRead("File-0027.AFS"))
{
//Read amount of files from offset 4
fs.Seek(4, SeekOrigin.Begin);
FileAmount = fs.ReadByte();
string strNumber = Convert.ToString(FileAmount);
fileamountStatus.Text = strNumber;
//Seek to beginning of LBA table
fs.Seek(8, SeekOrigin.Begin);
CurrentOffset = fs.Position;
int numBytesRead = 0;
while (Loop < FileAmount) //We want this to loop till it reachs our FileAmount number
{
Loop = Loop + 1;
//fread(&Start, 1, 4, ReadMunge); //Read File position
//Start = fs.ReadByte();
//Size = fs.ReadByte();
CurrentOffset = fs.Position;
int CurrentOffsetINT = unchecked((int)CurrentOffset);
//Start = fs.Read(bytes,0, 4);
Start = fs.Read(bytes, CurrentOffsetINT, 4);
Size = fs.Read(bytes, CurrentOffsetINT, 4);
Start = fs.ReadByte();
}
}
The problem I keep running into is that Start/Size do not hold the 4 bytes of data that I need.

If you're reading a binary file, you probably should look into using BinaryReader. That way you don't have to worry about converting byte arrays to integers or whatever. You can simply call reader.ReadInt32, for example, to read an int.