I am learning C#, and stack on BinaryReader/Writer. How i can read/write file in this cause:
I am need to store this data into binary file. (Sprite class)
public class Sprite
{
public Int32 id; //spriteID
public Int32 size; //sprite byte size
public byte[] dump; //sprite image byte-array
public Sprite()
{
id = 0;
size = 0;
dump = null;
}
}
I am get the sprite data by this function:
public Sprite createSprite(Image image, System.Drawing.Imaging.ImageFormat format)
{
using (MemoryStream ms = new MemoryStream())
{
Sprite tmpSpr = new Sprite();
image.Save(ms, format);
tmpSpr.dump = ms.ToArray();
tmpSpr.size = tmpSpr.Length;
tmpSpr.id = 1; //just for example
return tmpSpr;
}
}
But i am stack on solve this:
public static bool save(string filename)
{
FileStream fileStream = new FileStream(filename, FileMode.Create);
MemoryStream ms = new MemoryStream();
BinaryWriter wr = new BinaryWriter(ms);
//How i am should write a sprites here?
}
Here the unfinished reader function. Stack on reading too.
public static bool load(string filename, ref Dictionary<UInt32, Sprite> sprites)
{
FileStream fileStream = new FileStream(filename, FileMode.Open);
try
{
using (BinaryReader reader = new BinaryReader(fileStream))
{
UInt32 totalSprites = reader.ReadUInt32(); //total sprites in file
//something like that?
List<UInt32> spriteIndexes = new List<UInt32>();
for (uint i = 0; i < totalSprites; ++i)
{
UInt32 index = reader.ReadUInt32();
spriteIndexes.Add(index);
}
UInt32 id = 1;
foreach (UInt32 element in spriteIndexes)
{
//i am not sure here =(
reader.BaseStream.Seek(element, SeekOrigin.Begin);
UInt32 size = reader.ReadUInt32();
Sprite sprite;
if (sprites.TryGetValue(id, out sprite))
{
if (sprite != null && size > 0)
{
if (sprite.size > 0)
{
//generate warning
}
else
{
sprite.id = id;
sprite.size = size;
sprite.dump = reader.ReadBytes(size);
sprites[id] = sprite;
}
}
}
else
{
//i am not shure here too.
reader.BaseStream.Seek(size, SeekOrigin.Current);
}
++id;
}
}
}
finally
{
fileStream.Close();
}
return true;
}
The Sprite file, should have a totalSprites param, and list of all sprites, with their id, size, dump. Load function "seems" almost "done", but i dont have idea, how to write file for that "reader". Please, show the solution. Thanks for advance!
Your basic problem is that you don't know what the sprite's position in the file will be until you write it (unless you go through the array and count the bytes, which is possible), but you want to write the index ahead of the sprites. You have a few options:
First, why have an index at all? If you know how many sprites you have, you can just write that number and then read the file sequentially, one sprite at a time. That's the easiest:
using (var fs = File.OpenWrite(filename))
{
using (var writer = new BinaryWriter(fs))
{
// write sprite count
writer.Write(spriteList.Count);
// and write each sprite
foreach (var s in spriteList)
{
writer.Write(sprite.id);
writer.Write(sprite.size);
writer.Write(sprite.dump);
}
}
}
Reading is just the reverse: read the count, and then read that many sprites. There's no real need for an index unless you want the ability to read the fifth sprite, for example, without having to read the first four ahead of it.
If you really need the index, you can write a placeholder for it, then write the sprites, keeping track of their positions, and then seek back to the beginning of the file and write the real index. Like this:
var index = new List<long>(spriteList.Count);
using (var fs = File.OpenWrite(filename))
{
using (var writer = new BinaryWriter(fs))
{
// write sprite count
writer.Write(spriteList.Count);
var indexPos = writer.BaseStream.Position;
// write index placeholder
for (var i = 0; i < spriteList.Count; ++i)
{
writer.Write(0L);
}
// and write each sprite
for (var i = 0; i < spriteList.Count ++i)
{
// save current position
writer.Flush();
index[i] = writer.BaseStream.Position;
writer.Write(sprite.id);
writer.Write(sprite.size);
writer.Write(sprite.dump);
}
// Seek back to the index position
writer.Flush();
writer.BaseStream.Position = indexPos;
// and write the real index
foreach (var pos in index)
{
writer.Write(pos);
}
}
}
Another alternative is to write the sprites first and keep track of their positions in the index array as above, but write the index after the sprites. Then, to read, you read the sprite count from the start of the file, multiply that by 8 (the size of a long), and seek to the end of the file minus that much. That is:
var spriteCount = reader.ReadInt32();
long indexSize = spriteCount * 8;
reader.BaseStream.Seek(-indexSize, SeekOrigin.End);
// now read the index of spriteCount values
Related
I am trying to read objects from very large files containing padded structs that were written into it by a C++ process. I was using an example to memory map the large file and try to deserialize the data into an object but I now can see that it won't work this way.
How can I extract all the objects from the files to use in C#? I'm probably way off but I've provided the code. The objects have a 8 byte milliseconds member followed by 21 16bit integers, which needs 6bytes of padding to align to a 8byte boundary.
[Serializable]
unsafe public struct DataStruct
{
public UInt64 milliseconds;
[MarshalAs(UnmanagedType.ByValArray, SizeConst = 21)]
public fixed Int16 data[21];
[MarshalAs(UnmanagedType.ByValArray, SizeConst = 3)]
public fixed Int16 padding[3];
};
[Serializable]
public class DataArray
{
public DataStruct[] samples;
}
public static class Helper
{
public static Int16[] GetData(this DataStruct data)
{
unsafe
{
Int16[] output = new Int16[21];
for (int index = 0; index < 21; ++index)
output[index] = data.data[index];
return output;
}
}
}
class FileThreadSupport
{
struct DataFileInfo
{
public string path;
public UInt64 start;
public UInt64 stop;
public UInt64 elements;
};
// Create our epoch timestamp
private static readonly DateTime epoch = new DateTime(1970, 1, 1, 0, 0, 0, DateTimeKind.Utc);
// Output TCP client
private Support.AsyncTcpClient output;
// Directory which contains our data
private string replay_directory;
// Files to be read from
private DataFileInfo[] file_infos;
// Current timestamp of when the process was started
UInt64 process_start = 0;
// Object from current file
DataArray current_file_data;
// Offset into current files
UInt64 current_file_index = 0;
// Offset into current files
UInt64 current_file_offset = 0;
// Run flag
bool run = true;
public FileThreadSupport(ref Support.AsyncTcpClient output, ref Engine.A.Information info, ref Support.Configuration configuration)
{
// Set our output directory
replay_directory = configuration.getString("replay_directory");
if (replay_directory.Length == 0)
{
Console.WriteLine("Configuration does not provide a replay directory");
return;
}
// Check the directory for playable files
if(!loadDataDirectory(replay_directory))
{
Console.WriteLine("Replay directory {} did not have any valid files", replay_directory);
}
// Set the output TCP client
this.output = output;
}
private bool loadDataDirectory(string directory)
{
string[] files = Directory.GetFiles(directory, "*.*", SearchOption.TopDirectoryOnly);
file_infos = new DataFileInfo[files.Length];
int index = 0;
foreach (string file in files)
{
string[] parts = file.Split('\\');
string name = parts.Last();
parts = name.Split('.');
if (parts.Length != 2)
continue;
UInt64 start, stop = 0;
if (!UInt64.TryParse(parts[0], out start) || !UInt64.TryParse(parts[1], out stop))
continue;
long size = new System.IO.FileInfo(file).Length;
// Add to our file info array
file_infos[index] = new DataFileInfo
{
path = file,
start = start,
stop = stop,
elements = (ulong)(new System.IO.FileInfo(file).Length / 56
/*System.Runtime.InteropServices.Marshal.SizeOf(typeof(DataStruct))*/)
};
++index;
}
// Sort the array
Array.Sort(file_infos, delegate (DataFileInfo x, DataFileInfo y) { return x.start.CompareTo(y.start); });
// Return whether or not there were files found
return (files.Length > 0);
}
public void start()
{
process_start = (ulong)DateTime.Now.ToUniversalTime().Subtract(epoch).TotalMilliseconds;
UInt64 num_samples = 0;
while(run)
{
// Get our samples and add it to the sample
DataStruct[] result = getData(100);
Engine.A.A message = new Engine.A.A();
for (int i = 0; i < result.Length; ++i)
{
Engine.A.Data sample = new Engine.A.Data();
sample.Time = process_start + num_samples * 4;
Int16[] signal_data = Helper.GetData(result[i]);
for(int e = 0; e < signal_data.Length; ++e)
sample.Value[e] = signal_data[e];
message.Signal.Add(sample);
++num_samples;
}
// Send out the websocket
this.output.SendAsync(message.ToByteArray());
// Sleep 100 milliseconds
Thread.Sleep(100);
}
}
public void stop()
{
run = false;
}
private DataStruct[] getData(UInt64 milliseconds)
{
if (file_infos.Length == 0)
return new DataStruct[0];
if (current_file_data == null)
{
current_file_data = ReadObjectFromMMF(file_infos[current_file_index].path) as DataArray;
if(current_file_data.samples.Length == 0)
return new DataStruct[0];
}
UInt64 elements_to_read = (UInt64) milliseconds / 4;
DataStruct[] result = new DataStruct[elements_to_read];
Array.Copy(current_file_data.samples, (int)current_file_offset, result, 0, (int) Math.Min(elements_to_read, file_infos[current_file_index].elements - current_file_offset));
while((UInt64) result.Length != elements_to_read)
{
current_file_index = (current_file_index + 1) % (ulong) file_infos.Length;
current_file_data = ReadObjectFromMMF(file_infos[current_file_index].path) as DataArray;
if (current_file_data.samples.Length == 0)
return new DataStruct[0];
current_file_offset = 0;
Array.Copy(current_file_data.samples, (int)current_file_offset, result, result.Length, (int)Math.Min(elements_to_read, file_infos[current_file_index].elements - current_file_offset));
}
return result;
}
private object ByteArrayToObject(byte[] buffer)
{
BinaryFormatter binaryFormatter = new BinaryFormatter(); // Create new BinaryFormatter
MemoryStream memoryStream = new MemoryStream(buffer); // Convert buffer to memorystream
return binaryFormatter.Deserialize(memoryStream); // Deserialize stream to an object
}
private object ReadObjectFromMMF(string file)
{
// Get a handle to an existing memory mapped file
using (MemoryMappedFile mmf = MemoryMappedFile.CreateFromFile(file, FileMode.Open))
{
// Create a view accessor from which to read the data
using (MemoryMappedViewAccessor mmfReader = mmf.CreateViewAccessor())
{
// Create a data buffer and read entire MMF view into buffer
byte[] buffer = new byte[mmfReader.Capacity];
mmfReader.ReadArray<byte>(0, buffer, 0, buffer.Length);
// Convert the buffer to a .NET object
return ByteArrayToObject(buffer);
}
}
}
Well for one thing you're not using that memory mapped file well at all, you're just sequentially reading it all in a buffer, which is both needlessly inefficient and much slower than if you simply opened the file to read normally. The selling point of memory mapped files is repeated random access and random updates backed by the OS's virtual memory paging.
And you definitely don't need to read the entire file in memory, since your data is so strongly structured. You know exactly how many bytes to read for a record: Marshal.SizeOf<DataStruct>().
Then you need to get rid of all that serialization noise. Again your data is strongly typed, just read it. Get rid of those fixed arrays and use regular arrays, you're already instructing the marshaller how to read them with MarshalAs attributes (good). That also gets rid of that helper function that just copies an array for some unknown reason.
Your reading loop is very simple: read the correct number of bytes for one entry, use Marshal.PtrToStructure to convert it to a readable structure and add it to a list to return at the end. Bonus points if you can use .Net Core and Unsafe.As or Unsafe.Cast.
Edit: and don't use object returns, you know exactly what you're returning, write it down.
I have DataWriter that has file storage stream already attached. how ever in particular case I want to first write data in memory so I can know size of bytes, and store size with data in writer.
How can I do that without creating two in memory buffers?
DataWriter writer; // writer is parameter passed from somewhere else.
using (var inMemory = new InMemoryRandomAccessStream())
{
// fill inMemory with data.
// ***Here*** How can I avoid this?
var buffer = new byte[checked((int)inMemory.Position)].AsBuffer();
inMemory.Seek(0);
await inMemory.ReadAsync(buffer, buffer.Length, InputStreamOptions.ReadAhead);
writer.WriteUInt32(buffer.Length); // write size
writer.WriteBuffer(buffer); // write data
}
As you can see I'm using two buffers, one is for memory stream, the other is ibuffer.
I don't know how to directly write inMemory contents into DataWriter which has filestorage stream already attached.
I had to write my own buffer stream in order to prevent duplicate buffer creation. though stream buffer internally works like list but it has benefits when list grows large.
internal sealed class BufferStream : IDisposable
{
private byte[] _array = Array.Empty<byte>();
private int _index = -1;
private const int MaxArrayLength = 0X7FEFFFFF;
public int Capacity => _array.Length;
public int Length => _index + 1;
public void WriteIntoDataWriterStreamAsync(IDataWriter writer)
{
// AsBuffer wont cause copy, its just wrapper around array.
if(_index >= 0) writer.WriteBuffer(_array.AsBuffer(0, _index));
}
public void WriteBuffer(IBuffer buffer)
{
EnsureSize(checked((int) buffer.Length));
for (uint i = 0; i < buffer.Length; i++)
{
_array[++_index] = buffer.GetByte(i);
}
}
public void Flush()
{
Array.Clear(_array, 0, _index);
_index = -1;
}
// list like resizing.
private void EnsureSize(int additionSize)
{
var min = additionSize + _index;
if (_array.Length <= min)
{
var newsize = (int) Math.Min((uint) _array.Length * 2, MaxArrayLength);
if (newsize <= min) newsize = min + 1;
Array.Resize(ref _array, newsize);
}
}
public void Dispose()
{
_array = null;
}
}
Then I can easily do this.
using (var buffer = new BufferStream())
{
// fill buffer
writer.WriteInt32(buffer.Length); // write size
buffer.WriteIntoDataWriterStream(writer); // write data
}
I'm using NAudio to open a wav file.
After I have used the SimpleCompressor class I also must do some normalizing the volume of the file to 0db, but I have no idea how to do that.
At the moment I have this:
string strCompressedFile = "";
byte[] WaveData = new byte[audio.Length];
SimpleCompressorStream Compressor = new SimpleCompressorStream(audio);
Compressor.Enabled = true;
if (Compressor.Read(WaveData, 0, WaveData.Length) > 0)
{
//doing the normalizing now
}
How can I get the volume from the new byte array WaveData and how can I change it?
In WaveData is the entire wav file including the file header.
You definitely can change individual sample value so that it fits maximum level:
string strCompressedFile = "";
byte[] WaveData = new byte[audio.Length];
SimpleCompressorStream Compressor = new SimpleCompressorStream(audio);
Compressor.Enabled = true;
byte maxLevel = 20;
if (Compressor.Read(WaveData, 0, WaveData.Length) > 0)
{
for (int i = 0; i < audio.Length; i++)
{
if (WaveData[i] > maxLevel)
{
WaveData[i] = maxLevel;
}
}
}
I've added a loop which iterates through all the samples and if it's value is higher that maxLevel we set it to maxLevel.
I'm working with large files , beginning from 10Gb. I'm loading the parts of the file in the memory for processing. Following code works fine for smaller files (700Mb)
byte[] byteArr = new byte[layerPixelCount];
using (FileStream fs = File.OpenRead(recFileName))
{
using (BinaryReader br = new BinaryReader(fs))
{
fs.Seek(offset, SeekOrigin.Begin);
for (int i = 0; i < byteArr.Length; i++)
{
byteArr[i] = (byte)(br.ReadUInt16() / 256);
}
}
}
After opening a 10Gb file, the first run of this function is OK. But the second Seek() throws an IO exception:
An attempt was made to move the file pointer before the beginning of the file.
The numbers are:
fs.Length = 11998628352
offset = 4252580352
byteArr.Length = 7746048
I assumed that GC didn't collect the closed fs reference before the second call and tried
GC.Collect();
GC.WaitForPendingFinalizers();
but no luck.
Any help is apreciated
I'm guessing it's because either your signed integer indexer or offset is rolling over to negative values. Try declaring offset and i as long.
//Offest is now long
long offset = 4252580352;
byte[] byteArr = new byte[layerPixelCount];
using (FileStream fs = File.OpenRead(recFileName))
{
using (BinaryReader br = new BinaryReader(fs))
{
fs.Seek(offset, SeekOrigin.Begin);
for (long i = 0; i < byteArr.Length; i++)
{
byteArr[i] = (byte)(br.ReadUInt16() / 256);
}
}
}
My following written code logic is appropriate with large files beyond 4GB. The key issue to notice is the LONG data type used with the SEEK method. As a LONG is able to point beyond 2^32 data boundaries. In this example, the code is processing first processing the large file in chunks of 1GB, after the large whole 1GB chunks are processed, the left over (<1GB) bytes are processed. I use this code with calculating the CRC of files beyond the 4GB size. (using https://crc32c.machinezoo.com/ for the crc32c calculation in this example)
private uint Crc32CAlgorithmBigCrc(string fileName)
{
uint hash = 0;
byte[] buffer = null;
FileInfo fileInfo = new FileInfo(fileName);
long fileLength = fileInfo.Length;
int blockSize = 1024000000;
decimal div = fileLength / blockSize;
int blocks = (int)Math.Floor(div);
int restBytes = (int)(fileLength - (blocks * blockSize));
long offsetFile = 0;
uint interHash = 0;
Crc32CAlgorithm Crc32CAlgorithm = new Crc32CAlgorithm();
bool firstBlock = true;
using (FileStream fs = new FileStream(fileName, FileMode.Open, FileAccess.Read))
{
buffer = new byte[blockSize];
using (BinaryReader br = new BinaryReader(fs))
{
while (blocks > 0)
{
blocks -= 1;
fs.Seek(offsetFile, SeekOrigin.Begin);
buffer = br.ReadBytes(blockSize);
if (firstBlock)
{
firstBlock = false;
interHash = Crc32CAlgorithm.Compute(buffer);
hash = interHash;
}
else
{
hash = Crc32CAlgorithm.Append(interHash, buffer);
}
offsetFile += blockSize;
}
if (restBytes > 0)
{
Array.Resize(ref buffer, restBytes);
fs.Seek(offsetFile, SeekOrigin.Begin);
buffer = br.ReadBytes(restBytes);
hash = Crc32CAlgorithm.Append(interHash, buffer);
}
buffer = null;
}
}
//MessageBox.Show(hash.ToString());
//MessageBox.Show(hash.ToString("X"));
return hash;
}
I have a class in a dll which parses a file and returns a Stream which represents an FAT image (or any other)
My problem is when there is any other image the class creates about 3702 (on average) null bytes at the beginning of the stream.
So I have to edit the stream first and then save it to a file.
I have a code already but it works slow.
[Note : fts is the returned FileStream.]
BufferedStream bfs = new BufferedStream(fts);
BinaryReader bbr = new BinaryReader(bfs);
byte[] all_bytes = bbr.ReadBytes((int)fts.Length);
List<byte> nls = new List<byte>();
int index = 0;
foreach (byte bbrs in all_bytes)
{
if (bbrs == 0x00)
{
index++;
nls.Add(bbrs);
}
else
{
break;
}
}
byte[] nulls = new byte[nls.Count];
nulls = nls.ToArray();
//File.WriteAllBytes(outputDir + "Nulls.bin", nulls);
long siz = fts.Length - index;
byte[] file = new byte[siz];
bbr.BaseStream.Position = index;
file = bbr.ReadBytes((int)siz);
bbr.Close();
bfs.Close();
fts.Close();
bfs = null;
fts = null;
fts = new FileStream(outputDir + "Image.bin", FileMode.Create, FileAccess.Write);
bfs = new BufferedStream(fts);
bfs.Write(file, 0, (int)siz);
bfs.Close();
fts.Close();
Now, my question is :
How can I remove the nulls more efficiently and faster than the above code?
Instead of pushing bytes onto a List you could simply loop through your stream until you find the first non-null byte and then just copy the array from there using Array.Copy.
I would think about something like this (untested code):
int index = 0;
int currByte = 0;
while ((currByte = bbrs.ReadByte()) == 0x00)
{
index++;
}
// now currByte and everything to the end of the stream are the bytes you want.