We Keep Coding

Efficiently write ReadOnlySequence to Stream

Reading data using a PipeReader returns a ReadResult containing the requested data as a ReadOnlySequence<byte>. Currently I am using this (simplified) snippet to write the data fetched from the reader to my target stream:
var data = (await pipeReader.ReadAsync(cancellationToken)).Buffer;
// lots of parsing, advancing, etc.
var position = data.GetPosition(0);
while (data.TryGet(ref position, out var memory))
{
await body.WriteAsync(memory);
}
This seems to be a lot of code for such a basic task, which I would usually expect to be a one-liner in .NET. Analyzing the overloads provided by Stream I fail to see how this functionality can be achieved with less code.
Is there some kind of extension method I am missing?

Looks like what you are looking for is planned for .NET 7:
Stream wrappers for more types. Developers have asked for the ability to create a Stream around the contents of a ReadOnlyMemory, a ReadOnlySequence...
Not the greatest answer, but at least you can stop worrying that you are missing something obvious.

Pipeline pattern and disposable objects

Recently I started to investigate a Pipeline pattern or also known as Pipes and Filters. I thought it is a good way to structure the code and applications which just process the data.
I used this article as a base for my pipeline and steps implementation (but this is not so important).
As usual blog is covering simple scenario but in my case I need (or maybe not) to work on IDisposable objects which might travel through the process.
For instance Streams
Let's consider simple pipeline which should load csv file and insert its rows into some db. In simple abstraction we could implement such functions
Stream Step1(string filePath)
IEnumerable<RowType> Step2(Stream stream)
bool Step3(IEnumerable<RowType> data)
Now my question is if that is a good approach. Because if we implement that as step after step processing the Stream object leaves first step and it is easy to fall into a memory leakage problem.
I know that some might say that I should have Step1 which is loading and deserialising data but we are considering simple process. We might have more complex ones where passing a Stream makes more sense.
I am wondering how can I implement such pipelines to avoid memory leaks and also avoiding loading whole file into MemoryStream (which would be safer). Should I somehow wrap each step in try..catch blocks to call Dispose() if something goes wrong? Or should I pass all IDisposable resources into Pipeline object which will be wrapped with using to dispose all resources produced during processing correctly?

If it's planned to be used like Step3( Step2( Step1(filePath) ) ), then
Step2 should dispose the stream. It may use yield return feature of c#, which creates an implementation of IEnumerator<> underneath, which implements IDisposable, and allows for "subscribing" for the "event" of the finishing of enumerating and call Stream.Dispose at that point. E.g. :
IEnumerable<RowType> Step2(Stream stream)
{
using(stream)
using(StreamReader sr = new StreamReader(stream))
{
while(!sr.EndOfStream)
{
yield return Parse(sr.ReadLine()); //yield return implements IEnumerator<>
}
} // finally part of the using will be called from IEnumerator<>.Dispose()
}
Then if Step3 either uses LINQ
bool Step3(IEnumerable<RowType> data) => data.Any(item => SomeDecisionLogic(item));
or foreach
bool Step3(IEnumerable<RowType> data)
{
foreach(var item in data)
if(SomeDecisionLogic(item)))
return true;
}
for enumerating, both of them guarantee to call IEnumerator<>.Dispose() (ref1, ECMA-334 C# Spec, ch.13.9.5 ), which will call Stream.Dispose

IMO it's worth having a pipeline if the interaction is between at least 2 different systems and if the work can be executed in parallel. Otherwise it's more overhead.
In this case there are 2 systems: the file system where the CSV file is and the database. I think the pipeline should have at least 2 steps that run in parallel:
IEnumerable<Row> ReadFromCsv(string csvFilePath)
void UpdateDabase<IEnumerable<Row> rows)
In this case it should be clear that the Stream is bound to ReadFromCsv.
IEnumerable<Row> ReadFromCsv(path)
{
using(var stream = File.OpenRead(path))
{
var lines = GetLines(stream); // yield one at a time, not all at once
foreach (var line in line) yield return GetRow(line);
}
}
I guess the scope depends on the steps - which in turn depend on the way you design the pipeline based on your needs.

Can I use multiple BinaryWriters on the same Stream?

Can I create a new BinaryWriter and write on a Stream, while the stream is already beeing used by another BinaryWriter?
I need to write some data recursively, but I would like to avoid passing a BinaryWriter to a method as a parameter, as I need to pass a Stream instead. So, each method that will write data on the stream may need to create its own BinaryWriter instance. But I don't know if this is right. For now, it works well on a FileStream, but I don't know if it could lead to unexpected results on the users machines.
I wrote a simple example of what I want to achieve. Is this use of the BinaryWriter wrong?
Example:
public Main()
{
using (var ms = new MemoryStream())
{
// Write data on the stream.
WriteData(ms);
}
}
private void WriteData(Stream output)
{
// Create and use a BinaryWriter to use only on this method.
using (var bWriter = new BinaryWriter(output, Encoding.UTF8, true))
{
// Write some data using this BinaryWriter.
bWriter.Write("example data string");
// Send the stream to other method and write some more data there.
WriteMoreData(output);
// Write some more data using this BinaryWriter.
bWriter.Write("another example data string");
}
}
private void WriteMoreData(Stream output)
{
// Create and use a BinaryWriter to use only on this method.
using (var bWriter = new BinaryWriter(output, Encoding.Unicode, true))
{
// Write some data on this BinaryWriter.
bWriter.Write("write even more example data here");
}
}

Is this use of the BinaryWriter wrong?
It should work fine. BinaryWriter does no buffering itself, so each instance won't interfere with data written by other instances. You're passing true for the leaveOpen parameter, so when each instance is disposed, it won't close the underlying stream.
But "wrong" is to some degree in the eye of the beholder. I would say it's better to pass the BinaryWriter.
MemoryStream isn't buffered, but other types are. Each instance of BinaryWriter, when it's disposed, will flush the stream. This could be considered inefficient by some people, as it negates the benefit of the buffering, at least partially. Not an issue here, but may not be the best habit to get into.
In addition, each instance of BinaryWriter is going to create additional work for the garbage collector. If there's really only a few, that's probably not an issue. But if the real-world example involves a lot more calls, that could start to get noticeable, especially when the underlying stream is a MemoryStream (i.e. you're not dealing with some slow device I/O).
More to the point, I don't see any clear advantage to using multiple BinaryWriter instances on the same stream here. It seems like the natural, readable, easily-maintained thing to do would be to create a single BinaryWriter and reuse it until you're done writing.
Why do you want to avoid passing it as a parameter? You're already passing the Stream. Just pass the BinaryWriter instead. If you ever did need direct access to the underlying stream, it's always available via BinaryWriter.BaseStream.
Bottom line: I can't say there's anything clearly wrong per se with your proposal. But it's a deviation from normal conventions without (to me, anyway) a clear benefit. If you have a really good rationale for doing it this way, it should work. But I'd recommend against it.

Serialization in Unity

So to put it as briefly as possible, I am trying to save my character's essential information, and not only have I heard PlayerPrefs is ill-advised, but it also won't work right for some of the information I have (for instance, I can't PlayerPrefs my Profession and its accompanying stats and inherited class info), so I've pretty much assumed the best, if not only way, to accomplish this is through Serialization.
Now, I am pretty positive I understand Serialization in a core way, but I wouldn't claim that I know it very intimately, and thus, I'm in a bit of a bind.
I have quite a few scripts written, and here's the gist for them.
Note: My scripts very well may be a mess, but if that's so, please tell me. I don't claim that they're great, only that I have a lot there, and AFAIK, they're all alright, it's just doing the Serializing that is difficult for me for whatever reason.
Slight description of them: I am simply trying to make a character script for a Guard that will take both the Job: Mercenary, as well as the Type: GuardPrototype, and then, I want to be able to save that. In theory, the GameControl.cs script would accomplish that, but I'm having troubles (obviously), and I have a bunch of things commented out because I am fairly clueless, lol.
So, that said, I did do the Persistence and Saving tutorial from Unity, but I'm not only using/calling different scripts, I'm not handling simple floats, so I've had a hard time modifying that. Ultimately, I just want to know two things: Is my code that I am trying to save sensible? If it is, then how on earth would I use Serialization to save the info?
Thanks in advance, I appreciate any help I get.
TL;DR How in the hell does Serialization work with things that aren't simple floats, that are in separate scripts?
Notes:
The following are the chains of scripts I intend to use
ClassParadigm -> Mercenary //this is the job that gets used
TypeParadigm //because there are multiple it could be -> StandardParadigm -> GuardPrototype //of all the standard types, it is of a guard
Then, I want to have a script call them.
- Character (in this case, GuardA), which will then take a job, and a type (both established above), as well as StandardPlayerParadigm //What a standard player will possess
Finally, this is all supposed to be placed on an object in Unity, which I could then make a prefab of. So, in other words, if this were a character in my game, whenever that prefab was on the field, it'd be a GuardPrototype + Mercenary.
Edit: Thanks to Mike Hunt because they definitely assisted me big time with the main problem at hand. I now have a slightly different issue, but this seems MUCH more feasible.
Firstly, I updated my gist.
Secondly, I am having a thing in Unity where, when I attach the XMLSerialization script to a gameObject, it has some child profiles in it (like a nested menu that I don't want it to have).
I'm not quite sure how to combat that, and what's more it certainly doesn't seem like it's actually assigning the values I want it to have due to that (As in I want the GuardA script to have assigned stats from its "type" script I wrote, but I don't think it's working). I'm positive I just did something a bit excessive and somewhere in my code it made it call something extra, but I can't for the life of me figure out where that would've been.
So two questions now: A) What is going on with that?
B) Is this an effective use? Did I not quite implement this as intended?
Also, third question: This seems like an impeccable method for having duplicate enemies with minor variance in stats, but what exactly would I need to do to just save my standard player? Seems like it's still not quite hitting the mark for that, but I could be wrong and just not realize it.

If you want to use binary serialization that'll be the best to implement ISerializable.
You need provide 2 items:
Method for serialization, that will guide serializer 'what' and 'how' to save:
void GetObjectData(SerializationInfo info,
StreamingContext context)
Custom constructor. The ISerializable interface implies a constructor with the signature constructor
(SerializationInfo information, StreamingContext context)
And if you need another example article Object serizalization.
But for the game I would suggest to look at some custom Xml based serizaliser, so you don't need to write directions for binary serizaliation on every class change, and only properties needed. Ofc there might be some troubles with properties in Unity :(.

Create a class that will store the info to save and decorate it of the Serializable attribute:
[Serializable]
public class Storage{
public string name;
public int score;
}
When you want to save data, create an instance of this class, populate it, use .NET serialization and save with PlayerPrefs:
// Create
Storage storage = new Storage();
// Populate
storage.name = "Geoff";
storage.score = 10000;
// .NET serialization
BinaryFormatter bf = new BinaryFormatter();
MemoryStream ms = new MemoryStream();
bf.Serialize(ms, items as Storage);
// use PlayerPrefs
PlayerPrefs.SetString("data", Convert.ToBase64String(ms.GetBuffer()));
You can retrieve with the invert process:
if (PlayerPrefs.HasKey("data") == false) { return null; }
string str = PlayerPrefs.GetString("data");
BinaryFormatter bf = new BinaryFormatter();
MemoryStream ms = new MemoryStream(Convert.FromBase64String(str));
Storage storage = bf.Deserialize(ms) as Storage;
I would suggest to convert that into a generic method so you can use any type with any key:
public static class Save
{
public static void SaveData<T>(string key, object value) where T : class
{
BinaryFormatter bf = new BinaryFormatter();
MemoryStream ms = new MemoryStream();
bf.Serialize(ms, value as T);
PlayerPrefs.SetString(key, Convert.ToBase64String(ms.GetBuffer()));
}
public static T GetData<T>(string key) where T: class
{
if (PlayerPrefs.HasKey(key) == false) { return null; }
string str = PlayerPrefs.GetString(key);
BinaryFormatter bf = new BinaryFormatter();
MemoryStream ms = new MemoryStream(Convert.FromBase64String(str));
return bf.Deserialize(ms) as T;
}
}

.Net streams: Returning vs Providing

I have always wondered what the best practice for using a Stream class in C# .Net is. Is it better to provide a stream that has been written to, or be provided one?
i.e:
public Stream DoStuff(...)
{
var retStream = new MemoryStream();
//Write to retStream
return retStream;
}
as opposed to;
public void DoStuff(Stream myStream, ...)
{
//write to myStream directly
}
I have always used the former example for sake of lifecycle control, but I have this feeling that it a poor way of "streaming" with Stream's for lack of a better word.

I would prefer "the second way" (operate on a provided stream) since it has a few distinct advantages:
You can have polymorphism (assuming as evidenced by your signature you can do your operations on any type of Stream provided).
It's easily abstracted into a Stream extension method now or later.
You clearly divide responsibilities. This method should not care on how to construct a stream, only on how to apply a certain operation to it.
Also, if you're returning a new stream (option 1), it would feel a bit strange that you would have to Seek again first in order to be able to read from it (unless you do that in the method itself, which is suboptimal once more since it might not always be required - the stream might not be read from afterwards in all cases). Having to Seek after passing an already existing stream to a method that clearly writes to the stream does not seem so awkward.

I see the benefit of Streams is that you don't need to know what you're streaming to.
In the second example, your code could be writing to memory, it could be writing directly to file, or to some network buffer. From the function's perspective, the actual output destination can be decided by the caller.
For this reason, I would prefer the second option.
The first function is just writing to memory. In my opinion, it would be clearer if it did not return a stream, but the actual memory buffer. The caller can then attach a Memory Stream if he/she wishes.
public byte[] DoStuff(...)
{
var retStream = new MemoryStream();
//Write to retStream
return retStream.ToArray();
}

100% the second one. You don't want to make assumptions about what kind of stream they want. Do they want to stream to the network or to disk? Do they want it to be buffered? Leave these up to them.
They may also want to reuse the stream to avoid creating new buffers over and over. Or they may want to stream multiple things end-to-end on the same stream.
If they provide the stream, they have control over its type as well as its lifetime. Otherwise, you might as well just return something like a string or array. The stream isn't really giving you any benefit over these.