So I haven't written the code I'm dealing right now and I'm looking how to best handle this.
Right now I have this.
public static void WriteSymbol(Stream stream, Symbol symbol)
{
using (var streamWriter = new StreamWriter(stream))
{
JsonSerializer.Create(SerializerSettings).Serialize(streamWriter, symbol);
}
}
I'd like to be able to read the content of the stream after this is done in my test so that I can check the integration. The problem is that right now after Serialize the stream is closed and I can't read anymore from it.
I see that JsonWriter public bool CloseOutput { get; set; } but I don't see something for this static JsonSerializer.
What would be the best way to go about this?
How do I prevent the Serializer from closing the stream?
Is there some way I should check the content of the Stream?
From .net 4.5 upwards you can use the LeaveOpen constructor argument of StreamWriter.
The default buffer size used by the StreamWriter is 1024 as visible when decompiling the type.
So you can do the following.
using (var streamWriter = new StreamWriter(stream, Encoding.UTF8, 1024, true))
{
// TODO: do something
}
Try something like this:
public static void WriteSymbol(Stream stream, Symbol symbol)
{
using (var streamWriter = new StreamWriter(stream))
{
JsonSerializer.Create(SerializerSettings).Serialize(streamWriter, symbol);
// test stream here
}
}
Or don't surround this call with a using statement and then close the stream outside of this method.
Related
I'm having an issue with StreamWriter and Byte Order Marks. The documentation seems to state that the Encoding.UTF8 encoding has byte order marks enabled but when files are being written some have the marks while other don't.
I'm creating the stream writer in the following way:
this.Writer = new StreamWriter(this.Stream, System.Text.Encoding.UTF8);
Any ideas on what could be happening would be appreciated.
As someone pointed that out already, calling without the encoding argument does the trick.
However, if you want to be explicit, try this:
using (var sw = new StreamWriter(this.Stream, new UTF8Encoding(false)))
To disable BOM, the key is to construct with a new UTF8Encoding(false), instead of just Encoding.UTF8Encoding. This is the same as calling StreamWriter without the encoding argument, internally it's just doing the same thing.
To enable BOM, use new UTF8Encoding(true) instead.
Update: Since Windows 10 v1903, when saving as UTF-8 in notepad.exe, BOM byte is now an opt-in feature instead.
The issue is due to the fact that you are using the static UTF8 property on the Encoding class.
When the GetPreamble method is called on the instance of the Encoding class returned by the UTF8 property, it returns the byte order mark (the byte array of three characters) and is written to the stream before any other content is written to the stream (assuming a new stream).
You can avoid this by creating the instance of the UTF8Encoding class yourself, like so:
// As before.
this.Writer = new StreamWriter(this.Stream,
// Create yourself, passing false will prevent the BOM from being written.
new System.Text.UTF8Encoding());
As per the documentation for the default parameterless constructor (emphasis mine):
This constructor creates an instance that does not provide a Unicode byte order mark and does not throw an exception when an invalid encoding is detected.
This means that the call to GetPreamble will return an empty array, and therefore no BOM will be written to the underlying stream.
My answer is based on HelloSam's one which contains all the necessary information.
Only I believe what OP is asking for is how to make sure that BOM is emitted into the file.
So instead of passing false to UTF8Encoding ctor you need to pass true.
using (var sw = new StreamWriter("text.txt", new UTF8Encoding(true)))
Try the code below, open the resulting files in a hex editor and see which one contains BOM and which doesn't.
class Program
{
static void Main(string[] args)
{
const string nobomtxt = "nobom.txt";
File.Delete(nobomtxt);
using (Stream stream = File.OpenWrite(nobomtxt))
using (var writer = new StreamWriter(stream, new UTF8Encoding(false)))
{
writer.WriteLine("HelloПривет");
}
const string bomtxt = "bom.txt";
File.Delete(bomtxt);
using (Stream stream = File.OpenWrite(bomtxt))
using (var writer = new StreamWriter(stream, new UTF8Encoding(true)))
{
writer.WriteLine("HelloПривет");
}
}
The only time I've seen that constructor not add the UTF-8 BOM is if the stream is not at position 0 when you call it. For example, in the code below, the BOM isn't written:
using (var s = File.Create("test2.txt"))
{
s.WriteByte(32);
using (var sw = new StreamWriter(s, Encoding.UTF8))
{
sw.WriteLine("hello, world");
}
}
As others have said, if you're using the StreamWriter(stream) constructor, without specifying the encoding, then you won't see the BOM.
Do you use the same constructor of the StreamWriter for every file? Because the documentation says:
To create a StreamWriter using UTF-8 encoding and a BOM, consider using a constructor that specifies encoding, such as StreamWriter(String, Boolean, Encoding).
I was in a similar situation a while ago. I ended up using the Stream.Write method instead of the StreamWriter and wrote the result of Encoding.GetPreamble() before writing the Encoding.GetBytes(stringToWrite)
I found this answer useful (thanks to #Philipp Grathwohl and #Nik), but in my case I'm using FileStream to accomplish the task, so, the code that generates the BOM goes like this:
using (FileStream vStream = File.Create(pfilePath))
{
// Creates the UTF-8 encoding with parameter "encoderShouldEmitUTF8Identifier" set to true
Encoding vUTF8Encoding = new UTF8Encoding(true);
// Gets the preamble in order to attach the BOM
var vPreambleByte = vUTF8Encoding.GetPreamble();
// Writes the preamble first
vStream.Write(vPreambleByte, 0, vPreambleByte.Length);
// Gets the bytes from text
byte[] vByteData = vUTF8Encoding.GetBytes(pTextToSaveToFile);
vStream.Write(vByteData, 0, vByteData.Length);
vStream.Close();
}
Seems that if the file already existed and didn't contain BOM, then it won't contain BOM when overwritten, in other words StreamWriter preserves BOM (or it's absence) when overwriting a file.
Could you please show a situation where it don't produce it ? The only case where the preamble isn't present that I can find is when nothing is ever written to the writer (Jim Mischel seem to have find an other, logical and more likely to be your problem, see it's answer).
My test code :
var stream = new MemoryStream();
using(var writer = new StreamWriter(stream, System.Text.Encoding.UTF8))
{
writer.Write('a');
}
Console.WriteLine(stream.ToArray()
.Select(b => b.ToString("X2"))
.Aggregate((i, a) => i + " " + a)
);
After reading the source code of SteamWriter, you need to make sure you are creating a new file, then the byte order mark will add to the file.
https://github.com/dotnet/runtime/blob/6ef4b2e7aba70c514d85c2b43eac1616216bea55/src/libraries/System.Private.CoreLib/src/System/IO/StreamWriter.cs#L267
Code in Flush method
if (!_haveWrittenPreamble)
{
_haveWrittenPreamble = true;
ReadOnlySpan preamble = _encoding.Preamble;
if (preamble.Length > 0)
{
_stream.Write(preamble);
}
}
https://github.com/dotnet/runtime/blob/6ef4b2e7aba70c514d85c2b43eac1616216bea55/src/libraries/System.Private.CoreLib/src/System/IO/StreamWriter.cs#L129
Code set the value of _haveWrittenPreamble
// If we're appending to a Stream that already has data, don't
write
// the preamble.
if (_stream.CanSeek && _stream.Position > 0)
{
_haveWrittenPreamble = true;
}
using Encoding.Default instead of Encoding.UTF8 solved my problem
//C#
using (System.IO.StreamWriter writer =
new System.IO.StreamWriter("me00.txt", true))
{
writer.WriteLine("Hey"); //saved
}
System.IO.StreamWriter writer02 = new System.IO.StreamWriter("me01.txt", true);
writer02.WriteLine("Now hey x2"); //not saved
File me00.txt and me01.txt are created, however only the first file's content gets saved.
me00.txt will have line hey. me01.txt will be an empty txt file; "Now hey x2" is not saved.
What does the keyword "using" do to cause this observation?
You don't have to use "using". It's just a shortcut to keep you from doing even more typing...
The alternative is to nest the whole thing in a try-finally construct like the following:
System.IO.StreamWriter writer = null;
try
{
writer = new System.IO.StreamWriter("me00.txt", true);
writer.WriteLine("Hey");
}
finally
{
if (writer != null)
writer.Dispose();
)
When the writer is disposed, it is also closed, which is the step you're missing. The using presents a tidy way to do all of this compactly.
Flush the Buffer. It works with the using statement, because the buffer is flushed when the stream being disposed/closed.
Just call writer02.Flush();
From the MSDN on StreamWriter.Flush :
Flushing the stream will not flush its underlying encoder unless you explicitly call Flush or Close
Also, you need to dispose the Stream. Don't forget to call the Dispose method when you are done with the stream.
The using statement will do it for you. If you call Dispose, you might not need to call Flush.
Add:
writer02.Close();
--------------:
public void Test()
{
//C#
using (System.IO.StreamWriter writer = new System.IO.StreamWriter("me00.txt", true))
{
writer.WriteLine("Hey");
}
}
Because this code is identical to the code
public void Test()
{
//C#
System.IO.StreamWriter writer = null;
try
{
writer = new System.IO.StreamWriter("me01111.txt", true);
writer.WriteLine("Hey"); //saved
}
finally
{
if (writer != null) //Flush data
{
((IDisposable)writer).Dispose();
}
}
}
I use the following snippet of code, and I'm unsure whether I need to call the Flush methods (once on StreamWriter, once on MemoryStream):
//converts an xsd object to the corresponding xml string, using the UTF8 encoding
public string Serialize(T t)
{
using (var memoryStream = new MemoryStream())
{
var encoding = new UTF8Encoding(false);
using (var writer = new StreamWriter(memoryStream, encoding))
{
var serializer = new XmlSerializer(typeof (T));
serializer.Serialize(writer, t);
writer.Flush();
}
memoryStream.Flush();
return encoding.GetString(memoryStream.ToArray());
}
}
First of all, because the code is inside the using block, I think the automatically called dispose method might do this for me. Is this true, or is flushing an entirely different concept?
According to stackoverflow itself:
Flush meaning clears all buffers for a stream and causes any buffered data to be written to the underlying device.
What does that mean in the context of the code above?
Secondly, the flush method of the MemoryStream does nothing according to the api, so what's up with that? why do we call a method that does nothing?
You don't need to use Flush on the StreamWriter, as you are disposing it (by having it in a using block). When it's disposed, it's automatically flushed and closed.
You don't need to use Flush on the MemoryStream, as it's not buffering anything that is written to any other source. There is simply nothing to flush anywhere.
The Flush method is only present in the MemoryStream object because it inherits from the Stream class. You can see in the source code for the MemoryStream class that the flush method actually does nothing.
In general Streams will buffer data as it's written (periodically flushing the buffer to the associated device if there is one) because writing to a device, usually a file, is expensive. A MemoryStream writes to RAM so the whole concept of buffering and flushing is redundant. The data is always in RAM already.
And yes, disposing the stream will cause it to be flushed.
Commenting flush method returning empty byte[], Though I am Using Using block
byte[] filecontent = null;
using var ms = new MemoryStream();
using var sw = new StreamWriter(fs);
sw.WriteCSVLine(new[] { "A", "B" });//This is extension to write as CSV
//tx.Flush();
//fs.Flush();
fs.Position = 0;
filecontent = fs.ToArray();
I'm having an issue with StreamWriter and Byte Order Marks. The documentation seems to state that the Encoding.UTF8 encoding has byte order marks enabled but when files are being written some have the marks while other don't.
I'm creating the stream writer in the following way:
this.Writer = new StreamWriter(this.Stream, System.Text.Encoding.UTF8);
Any ideas on what could be happening would be appreciated.
As someone pointed that out already, calling without the encoding argument does the trick.
However, if you want to be explicit, try this:
using (var sw = new StreamWriter(this.Stream, new UTF8Encoding(false)))
To disable BOM, the key is to construct with a new UTF8Encoding(false), instead of just Encoding.UTF8Encoding. This is the same as calling StreamWriter without the encoding argument, internally it's just doing the same thing.
To enable BOM, use new UTF8Encoding(true) instead.
Update: Since Windows 10 v1903, when saving as UTF-8 in notepad.exe, BOM byte is now an opt-in feature instead.
The issue is due to the fact that you are using the static UTF8 property on the Encoding class.
When the GetPreamble method is called on the instance of the Encoding class returned by the UTF8 property, it returns the byte order mark (the byte array of three characters) and is written to the stream before any other content is written to the stream (assuming a new stream).
You can avoid this by creating the instance of the UTF8Encoding class yourself, like so:
// As before.
this.Writer = new StreamWriter(this.Stream,
// Create yourself, passing false will prevent the BOM from being written.
new System.Text.UTF8Encoding());
As per the documentation for the default parameterless constructor (emphasis mine):
This constructor creates an instance that does not provide a Unicode byte order mark and does not throw an exception when an invalid encoding is detected.
This means that the call to GetPreamble will return an empty array, and therefore no BOM will be written to the underlying stream.
My answer is based on HelloSam's one which contains all the necessary information.
Only I believe what OP is asking for is how to make sure that BOM is emitted into the file.
So instead of passing false to UTF8Encoding ctor you need to pass true.
using (var sw = new StreamWriter("text.txt", new UTF8Encoding(true)))
Try the code below, open the resulting files in a hex editor and see which one contains BOM and which doesn't.
class Program
{
static void Main(string[] args)
{
const string nobomtxt = "nobom.txt";
File.Delete(nobomtxt);
using (Stream stream = File.OpenWrite(nobomtxt))
using (var writer = new StreamWriter(stream, new UTF8Encoding(false)))
{
writer.WriteLine("HelloПривет");
}
const string bomtxt = "bom.txt";
File.Delete(bomtxt);
using (Stream stream = File.OpenWrite(bomtxt))
using (var writer = new StreamWriter(stream, new UTF8Encoding(true)))
{
writer.WriteLine("HelloПривет");
}
}
The only time I've seen that constructor not add the UTF-8 BOM is if the stream is not at position 0 when you call it. For example, in the code below, the BOM isn't written:
using (var s = File.Create("test2.txt"))
{
s.WriteByte(32);
using (var sw = new StreamWriter(s, Encoding.UTF8))
{
sw.WriteLine("hello, world");
}
}
As others have said, if you're using the StreamWriter(stream) constructor, without specifying the encoding, then you won't see the BOM.
Do you use the same constructor of the StreamWriter for every file? Because the documentation says:
To create a StreamWriter using UTF-8 encoding and a BOM, consider using a constructor that specifies encoding, such as StreamWriter(String, Boolean, Encoding).
I was in a similar situation a while ago. I ended up using the Stream.Write method instead of the StreamWriter and wrote the result of Encoding.GetPreamble() before writing the Encoding.GetBytes(stringToWrite)
I found this answer useful (thanks to #Philipp Grathwohl and #Nik), but in my case I'm using FileStream to accomplish the task, so, the code that generates the BOM goes like this:
using (FileStream vStream = File.Create(pfilePath))
{
// Creates the UTF-8 encoding with parameter "encoderShouldEmitUTF8Identifier" set to true
Encoding vUTF8Encoding = new UTF8Encoding(true);
// Gets the preamble in order to attach the BOM
var vPreambleByte = vUTF8Encoding.GetPreamble();
// Writes the preamble first
vStream.Write(vPreambleByte, 0, vPreambleByte.Length);
// Gets the bytes from text
byte[] vByteData = vUTF8Encoding.GetBytes(pTextToSaveToFile);
vStream.Write(vByteData, 0, vByteData.Length);
vStream.Close();
}
Seems that if the file already existed and didn't contain BOM, then it won't contain BOM when overwritten, in other words StreamWriter preserves BOM (or it's absence) when overwriting a file.
Could you please show a situation where it don't produce it ? The only case where the preamble isn't present that I can find is when nothing is ever written to the writer (Jim Mischel seem to have find an other, logical and more likely to be your problem, see it's answer).
My test code :
var stream = new MemoryStream();
using(var writer = new StreamWriter(stream, System.Text.Encoding.UTF8))
{
writer.Write('a');
}
Console.WriteLine(stream.ToArray()
.Select(b => b.ToString("X2"))
.Aggregate((i, a) => i + " " + a)
);
After reading the source code of SteamWriter, you need to make sure you are creating a new file, then the byte order mark will add to the file.
https://github.com/dotnet/runtime/blob/6ef4b2e7aba70c514d85c2b43eac1616216bea55/src/libraries/System.Private.CoreLib/src/System/IO/StreamWriter.cs#L267
Code in Flush method
if (!_haveWrittenPreamble)
{
_haveWrittenPreamble = true;
ReadOnlySpan preamble = _encoding.Preamble;
if (preamble.Length > 0)
{
_stream.Write(preamble);
}
}
https://github.com/dotnet/runtime/blob/6ef4b2e7aba70c514d85c2b43eac1616216bea55/src/libraries/System.Private.CoreLib/src/System/IO/StreamWriter.cs#L129
Code set the value of _haveWrittenPreamble
// If we're appending to a Stream that already has data, don't
write
// the preamble.
if (_stream.CanSeek && _stream.Position > 0)
{
_haveWrittenPreamble = true;
}
using Encoding.Default instead of Encoding.UTF8 solved my problem
I have a binary file to which I want to append a chunk of data at the end of the file, how can I achieve this using C# and .net? Also is there anything to consider when writing to the end of a binary file? Thanks a lot for your help.
private static void AppendData(string filename, int intData, string stringData, byte[] lotsOfData)
{
using (var fileStream = new FileStream(filename, FileMode.Append, FileAccess.Write, FileShare.None))
using (var bw = new BinaryWriter(fileStream))
{
bw.Write(intData);
bw.Write(stringData);
bw.Write(lotsOfData);
}
}
You should be able to do this via the Stream:
using (FileStream data = new FileStream(path, FileMode.Append))
{
data.Write(...);
}
As for considerations - the main one would be: does the underlying data format support append? Many don't, unless it is your own raw data, or text etc. A well-formed xml document doesn't support append (without considering the final end-element), for example. Nor will something like a Word document. Some do, however. So; is your data OK with this...
Using StreamWriter and referencing DotNetPerls, make sure to add the True boolean to the StreamWriter constructor, if otherwise left blank, it'll overwrite as usual:
using System.IO;
class Program
{
static void Main()
{
// 1: Write single line to new file
using (StreamWriter writer = new StreamWriter("C:\\log.txt", true))
{
writer.WriteLine("Important data line 1");
}
// 2: Append line to the file
using (StreamWriter writer = new StreamWriter("C:\\log.txt", true))
{
writer.WriteLine("Line 2");
}
}
}
Output
(File "log.txt" contains these lines.)
Important data line 1
Line 2
This is the solution that I was actually looking for when I got here from Google, although it wasn't a binary file though, hope it helps someone else.