We Keep Coding

FileStream.WriteAsync(buffer.AsMemory(0, read)) won't work when writing small files but CopyToAsync will

I have small sig files that are exactly 256 bytes. When uploading to a file upload controller on asp.net core web app, the buffer is occupied correctly for the 256 positions but they aren't written to the output stream and the file is empty. CopyToAsync works fine. This will only happen to certain files. The problem is reproducible on a console application:
string SoureFile = #"C:\Users\me\source\repos\files\mySigFile.sig";
byte[] buffer = new byte[1024 * 64];
string tmp = #"C:\Users\me\Downloads\tempsigfile.tmp";
string tmp2 = #"C:\Users\me\Downloads\tempsigfile2.tmp";
var inputStream = File.Open(SoureFile, FileMode.OpenOrCreate);
//doesn't work
using FileStream writer = new(tmp, FileMode.Create);
int read;
while ((read = await inputStream.ReadAsync(buffer)) != 0)
{
await writer.WriteAsync(buffer.AsMemory(0, read));
}
inputStream.Position = 0;
//works
using (var stream = new FileStream(tmp2, FileMode.Create))
{
await inputStream.CopyToAsync(stream);
}
FileInfo info = new FileInfo(tmp);
Console.WriteLine(info.Length); //0
FileInfo info2 = new FileInfo(tmp2);
Console.WriteLine(info2.Length);//256

Doing this (using declaration, no bracers):
using FileStream writer = new(tmp, FileMode.Create);
means writer will only be disposed at the end of the scope, so at the end of the method. Doing WriteAsync does not necessary mean that information will be written to file right away, it might be written to the internal in-memory buffer and only written to the file when this buffer fills or when you close the file, or when you explicitly call Flush on a stream. You don't do anything of that, the file is only closed at the end of the method, but your check:
FileInfo info = new FileInfo(tmp);
Console.WriteLine(info.Length); //0
is performed before the actual write to file happens, so you see 0. If you actually check the contents of the file after this method (program) completes - you'll see it contains the correct data.
In second case you use using statement:
using (var stream = new FileStream(tmp2, FileMode.Create))
{
await inputStream.CopyToAsync(stream);
}
so you write to a file, close it, and only after that check the contents. Then it works as you expect.

What is the efficient way to write stream of bytes into a file in c#?

I have big big data in form of bytes around 5GB.
I need to store this data in a file ServerData.xml. This data should be first converted into string and then should be saved to file so that we can perform operation on the file.
I used below code to convert stream of bytes to string and then to save the same in a file.
private const string fileName = "ServerData.xml";
public void ProcessBuffer(byte[] receiveBuffer, int bytes)
{
if (!File.Exists(fileName))
{
using (File.Create(fileName)) { };
}
TextWriter tw = new StreamWriter(fileName, true);
tw.Write(Encoding.UTF8.GetString(receiveBuffer).TrimEnd((Char)0));
tw.Close();
}
Is it the right way ?
or please suggest better way so that there should not be any memory issue if any in future ?

The code in your question can only work if ProcessBuffer is always called with a UTF-8 encoded text that is broken on code point boundaries. That seems pretty unlikely to me, so I would expect that you encounter errors when decoding to text.
However, decoding to text and then writing, is rather pointless and indeed counter-productive. The bytes are already UTF-8 encoded. Write them directly to file as they arrive from the socket. Don't perform any processing of them. When you come to read the XML using XmlReader, the parser will read the encoding as UTF-8 from the document's XML declaration, and be able to decode the rest of the document. I am assuming that the document's XML declaration specifies UTF-8 but that seems highly likely. You should check.
You should get rid of the text writer which is no use to you for writing bytes. Write the bytes directly to a file stream. And try to avoid opening and closing the file repeatedly. That's very inefficient. Open and close the file exactly once.

Why do you need to convert it to a string?
using System.IO;
public static void WriteBytes(byte[] bytes, string filename)
{
using (FileStream fs = new FileStream(filename, FileMode.OpenOrCreate))
using (BinaryWriter writer = new BinaryWriter(fs, Encoding.UTF8))
{
writer.Write(bytes);
}
}

You can simply write these bytes to a file using FileStream:
public void ProcessBuffer(byte[] receivedBuffer, int bytes)
{
using (var fileStream = new FileStream(fileName, FileMode.Create)) // overwrites file
{
fileStream.Write(receivedBuffer, 0, bytes);
}
}
Update: You won't be able to work with such a big XML document if you don't have enough resources. I would suggest reformatting this file. For example, I would parse this XML and insert data into a SQL database. Then, you can easily operate with such amounts of data.

I would prefer that I write all bytes to file. And when reading, convert it to string and then convert to XML using XDocument, XElement etc. By writing bytes in file you will save space, and it is efficient,
Instead of using FileStream, I will prefer File.WriteAllBytes method.
private const string fileName = "ServerData.xml";
public void ProcessBuffer(byte[] receiveBuffer, int bytes)
{
File.WriteAllBytes(filename, bytes);
// And when reading
var bytes = File.ReadAllBytes(filename);
var binaryReader = new BinaryReader(new MemoryStream(bytes));
// Parse strings and make xml,
binaryReader.ReadString();
}

Placing a string in memory and saving it later

I have a program that creates a huge string, too large for StringBuilder to handle. I want to store it in memory, then, at a later date, save it to a user defined directory.
I have tried a StreamWriter but I need to declare the directory as I create it and that is problem because I get the directory later on, if the user selects the "Save to text file".
Basically I need it to:
-Create instance of whatever it is (public and in a class scope, can not be in a method scope)
-Add a string to instance
-Then, later on, whenever the user decides, save that to a file. I am aware it will take up a large amount of RAM and I believe it will only be around 50-100Mb, which isn't too bad.
public class ImageProcessor
{
MemoryStream ms = new MemoryStream();
StreamWriter sw;
public Bitmap Rendering(string bmpPath)
{
sw = new StreamWriter(ms, System.Text.Encoding.UTF8);
using (ms)
sw.Write(Convert.ToString(ArGBformat)); //Error is here
}
}
`

Here's a code snippet that may help you. I just tested it in VS 2012:
using (MemoryStream ms = new MemoryStream)
{
StreamWriter sw = new StreamWriter(ms, System.Text.Encoding.UTF8)
sw.WriteLine("This is a test.");
sw.WriteLine("This is a second line.");
sw.Flush();
using (FileStream fs = new FileStream("Test.txt", FileMode.Create))
{
ms.CopyTo(fs);
}
sw.Close();
}
The file contents are:
This is a test.
This is a second line.
You'll want to modify this to fit your program's design, but the basic idea is using a StreamWriter to write the text to the MemoryStream, and then writing the MemoryStream to the file with the MemoryStream.CopyTo method (which takes a Stream).
Be careful with how you construct things - if you close the StreamWriter it will close the MemoryStream as well (the first time I tested this I had the StreamWriter in a using block inside the MemoryStream using block, and then got an error trying to access a closed stream).
Hopefully this will at least get you going in the right direction.
EDIT
You'll need to initialize the StreamWriter in a method, not as a field variable.
Something like this:
public class ImageProcessor
{
MemoryStream ms = new MemoryStream();
StreamWriter sw;
public Bitmap Rendering(string bmpPath)
{
sw = new StreamWriter(ms, System.Text.Encoding.UTF8);
}
}

string serialization and deserialization problem

I'm trying to serialize/deserialize string. Using the code:
private byte[] StrToBytes(string str)
{
BinaryFormatter bf = new BinaryFormatter();
MemoryStream ms = new MemoryStream();
bf.Serialize(ms, str);
ms.Seek(0, 0);
return ms.ToArray();
}
private string BytesToStr(byte[] bytes)
{
BinaryFormatter bfx = new BinaryFormatter();
MemoryStream msx = new MemoryStream();
msx.Write(bytes, 0, bytes.Length);
msx.Seek(0, 0);
return Convert.ToString(bfx.Deserialize(msx));
}
This two code works fine if I play with string variables.
But If I deserialize a string and save it to a file, after reading the back and serializing it again, I end up with only first portion of the string.
So I believe I have a problem with my file save/read operation. Here is the code for my save/read
private byte[] ReadWhole(string fileName)
{
try
{
using (BinaryReader br = new BinaryReader(new FileStream(fileName, FileMode.Open)))
{
return br.ReadBytes((int)br.BaseStream.Length);
}
}
catch (Exception)
{
return null;
}
}
private void WriteWhole(byte[] wrt,string fileName,bool append)
{
FileMode fm = FileMode.OpenOrCreate;
if (append)
fm = FileMode.Append;
using (BinaryWriter bw = new BinaryWriter(new FileStream(fileName, fm)))
{
bw.Write(wrt);
}
return;
}
Any help will be appreciated.
Many thanks
Sample Problematic Run:
WriteWhole(StrToBytes("First portion of text"),"filename",true);
WriteWhole(StrToBytes("Second portion of text"),"filename",true);
byte[] readBytes = ReadWhole("filename");
string deserializedStr = BytesToStr(readBytes); // here deserializeddStr becomes "First portion of text"

Just use
Encoding.UTF8.GetBytes(string s)
Encoding.UTF8.GetString(byte[] b)
and don't forget to add System.Text in your using statements
BTW, why do you need to serialize a string and save it that way?
You can just use File.WriteAllText() or File.WriteAllBytes. The same way you can read it back, File.ReadAllBytes() and File.ReadAllText()

The problem is that you are writing two strings to the file, but only reading one back.
If you want to read back multiple strings, then you must deserialize multiple strings. If there are always two strings, then you can just deserialize two strings. If you want to store any number of strings, then you must first store how many strings there are, so that you can control the deserialization process.
If you are trying to hide data (as indicated by your comment to another answer), then this is not a reliable way to accomplish that goal. On the other hand, if you are storing data an a user's hard-drive, and the user is running your program on their local machine, then there is no way to hide the data from them, so this is as good as anything else.

How to append data to a binary file?

I have a binary file to which I want to append a chunk of data at the end of the file, how can I achieve this using C# and .net? Also is there anything to consider when writing to the end of a binary file? Thanks a lot for your help.

private static void AppendData(string filename, int intData, string stringData, byte[] lotsOfData)
{
using (var fileStream = new FileStream(filename, FileMode.Append, FileAccess.Write, FileShare.None))
using (var bw = new BinaryWriter(fileStream))
{
bw.Write(intData);
bw.Write(stringData);
bw.Write(lotsOfData);
}
}

You should be able to do this via the Stream:
using (FileStream data = new FileStream(path, FileMode.Append))
{
data.Write(...);
}
As for considerations - the main one would be: does the underlying data format support append? Many don't, unless it is your own raw data, or text etc. A well-formed xml document doesn't support append (without considering the final end-element), for example. Nor will something like a Word document. Some do, however. So; is your data OK with this...

Using StreamWriter and referencing DotNetPerls, make sure to add the True boolean to the StreamWriter constructor, if otherwise left blank, it'll overwrite as usual:
using System.IO;
class Program
{
static void Main()
{
// 1: Write single line to new file
using (StreamWriter writer = new StreamWriter("C:\\log.txt", true))
{
writer.WriteLine("Important data line 1");
}
// 2: Append line to the file
using (StreamWriter writer = new StreamWriter("C:\\log.txt", true))
{
writer.WriteLine("Line 2");
}
}
}
Output
(File "log.txt" contains these lines.)
Important data line 1
Line 2
This is the solution that I was actually looking for when I got here from Google, although it wasn't a binary file though, hope it helps someone else.