We Keep Coding

Read from a compressing GZipStream

I'm exploring how to implement an HTTP server in C#. (And before you ask, I know there is Kestrel (and nothing else that isn't obsolete), and I want a much, much smaller application.) So, the response could be a Stream that cannot be seeked and has an unknown length. For this situation, chunked encoding can be used instead of sending a Content-Length header.
The response can also be compressed with gzip or br as indicated by the client. This can be accomplished with e.g. the GZipStream class. I had almost said "easily", because that's not really the case. I always find the GZipStream API confusing each time I use it. I usually bump into every exception there is until I finally get it right.
It seems like I can only write (push) to a GZipStream and the compressed data will trickle out the other end into the specified "base" stream. But that's not desirable because I can't just let the compressed data flow to the client. It needs to be chunked. That is, each bit of compressed data needs to be prefixed with its chunk size. Of course the GZipStream cannot produce that format.
Instead, I'd like to read (pull) from the compressing GZipStream, but that doesn't seem to be possible. The documentation says it will throw an exception if I try that. But there has to be some instance that brings the compressed bytes into the chunked format.
So how would I get the expected result? Can it even be achieved with this API? Why can't I pull from the compressing stream, only push?
I'm not trying to make up (non-functional) sample code because that would only be confusing.
PS: Okay, maybe this:
Stream responseBody = ...;
if (canCompress)
{
responseBody = new GZipStream(responseBody, CompressionMode.Compress); // <-- probably wrong
}
// not shown: add appropriate headers
while (true)
{
int chunkLength = responseBody.Read(buffer); // <-- not possible
if (chunkLength == 0)
break;
response.Write($"{chunkLength:X}\r\n");
response.Write(buffer.AsMemory()[..chunkLength]);
response.Write("\r\n");
}
response.Write("0\r\n\r\n");

Your usage of GZipStream is incomplete. While your input responseBuffer is the correct target buffer, you have to actually write the bytes TO the GZipStream itself.
In addition, once you are done writing, you must close the GZipStream instance to write all compressed bytes to your target buffer. This is the critical step because there is no such thing as "partial compression" of an input stream in GZip. You would have to analyze the entire input in order to properly compress it. As such, this is the critical missing link that MUST happen before you can continue to write the response.
Finally, you need to reset the position of your output stream so that you can read it into an intermediary response buffer.
using MemoryStream responseBody = new MemoryStream();
GZipStream gzipStream = null; // make sure to dispose after use
if (canCompress)
{
using MemoryStream gzipStreamBuffer = new MemoryStream(bytes);
gzipStream = new GZipStream(responseBody, CompressionMode.Compress, true);
gzipStreamBuffer.CopyTo(gzipStream);
gzipStream.Close(); // close the stream so that all compressed bytes are written
responseBody.Seek(0, SeekOrigin.Begin); // reset the response so that we can read it to the buffer
}
var buffer = new byte[20];
while (true)
{
int chunkLength = responseBody.Read(buffer);
if (chunkLength == 0)
break;
// write response
}
In my test example, my bytes input was 241 bytes, whereas the compressed bytes written to the buffer totaled 82 bytes.

How to use MP3Sharp with IFormFile

I want to convert an mp3 file to pcm using MP3Sharp (https://github.com/ZaneDubya/MP3Sharp) in a web app, where the mp3 file is passed in as IFormFile
It works if I save the file to disk first like this ...
using (Stream fileStream = new FileStream("file.mp3", FileMode.Create))
{
await file.CopyToAsync(fileStream);
}
MP3Stream stream = new MP3Stream("file.mp3");
... but when I try to do it as a stream without writing to file it doesn't work:
using (var fileStream = new MemoryStream())
{
await file.CopyToAsync(fileStream);
MP3Stream stream = new MP3Stream(fileStream);
}
The MP3Stream constructor throws this exception:
MP3Sharp.MP3SharpException: 'Unhandled channel count rep: -1 (allowed values are 1-mono and 2-stereo).'
... Any ideas on what I'm doing wrong?

After your code does file.CopyToAsync(fileStream), the MemoryStream's read/write position is pointing after the written MP3 data at the end of the MemoryStream.
The MP3Stream then trying to read from the MemoryStream will only notice that the end of the MemoryStream has been reached (because its read/write position is already at the end) and throws an exception.[1]
Thus, after copying the MP3 data into the MemoryStream, set the MemoryStream's read/write position back to where it was before you copied the MP3 data into it (in the case of your example code it's the beginning of the MemoryStream, position 0):
await file.CopyToAsync(fileStream);
fileStream.Position = 0;
Side note: Like FileStream and MemoryStream, MP3Stream is also a Stream and therefore an IDisposable, too. And like you already did with the FileStream and the MemoryStream, you should use the using statement for the MP3Stream as well.
[1] The exception you got from the MP3Sharp library is misleading, and as such is kind of a bug in the library. Because, when attempting to read a byte from a stream and the stream is at its end, the Stream.ReadByte method will return -1 to indicate end-of-stream. And as apparent by the exception message, it seems the MP3Sharp library does not properly treat the -1 value here as simply meaning "the end of the stream has been reached and no (further) data could be read" and misinterprets it as a channel count value.

Do I need to 'rewind' a stream before reading what was written to it?

With this code:
using (var stream = new MemoryStream())
{
thumbnail.Save(stream); // you get the idea
stream.Position = 0; // <- is this needed?
WriteStreamToDisk(stream);
}
If I have a method writing to a memory stream, and then I want to write that stream to disk, do I need to set the position to 0?
Or, do streams have different read / write pointers?

A stream has only a single position which is used for both reading and writing. So, assuming that...
Thumbnail.Save(O); doesn't rewind the stream after it's done writing to the stream, and
WriteStreamToDisk(O); doesn't rewind the stream before it starts reading from the stream,
then yes, you will need to rewind the stream yourself.

NancyFx: Are streams disposed when using StreamResponse()?

Are streams disposed in NancyFx when using StreamResponse()?
Using a streamWriter to write to the stream, I can't dispose of it without closing the stream I want to send out. And the way I see it, I can't dispose of the stream manually either since I effectively left the method at the return.
This code works:
// Open a stream and write to it with streamReader
Stream memStream = new MemoryStream();
StreamWriter memWriter = new StreamWriter(memStream, Encoding.UTF8);
memWriter.Write(someStringText);
memWriter.Flush();
memStream.Position = 0;
// Add stream to the respose which should get downloaded
var response = new StreamResponse(() => memStream, MimeTypes.GetMimeType(contentType));
return response.AsAttachment(someFileName);
Do I need to worry about the stream lingering on until the garbage collector comes along or do it get disposed correctly by the StreamResponse?

According to StreamResponse.cs, StreamResponse.Dispose() disposes of the stream returned by the Func<Stream>, and Nancy handles disposing the Response at the end of the NancyContext.
As for disposing of StreamWriter without affecting the underlying stream, see this answer.

Generate zip file with xml content on the fly [duplicate]

I want to write a String to a Stream (a MemoryStream in this case) and read the bytes one by one.
stringAsStream = new MemoryStream();
UnicodeEncoding uniEncoding = new UnicodeEncoding();
String message = "Message";
stringAsStream.Write(uniEncoding.GetBytes(message), 0, message.Length);
Console.WriteLine("This:\t\t" + (char)uniEncoding.GetBytes(message)[0]);
Console.WriteLine("Differs from:\t" + (char)stringAsStream.ReadByte());
The (undesired) result I get is:
This: M
Differs from: ?
It looks like it's not being read correctly, as the first char of "Message" is 'M', which works when getting the bytes from the UnicodeEncoding instance but not when reading them back from the stream.
What am I doing wrong?
The bigger picture: I have an algorithm which will work on the bytes of a Stream, I'd like to be as general as possible and work with any Stream. I'd like to convert an ASCII-String into a MemoryStream, or maybe use another method to be able to work on the String as a Stream. The algorithm in question will work on the bytes of the Stream.

After you write to the MemoryStream and before you read it back, you need to Seek back to the beginning of the MemoryStream so you're not reading from the end.
UPDATE
After seeing your update, I think there's a more reliable way to build the stream:
UnicodeEncoding uniEncoding = new UnicodeEncoding();
String message = "Message";
// You might not want to use the outer using statement that I have
// I wasn't sure how long you would need the MemoryStream object
using(MemoryStream ms = new MemoryStream())
{
var sw = new StreamWriter(ms, uniEncoding);
try
{
sw.Write(message);
sw.Flush();//otherwise you are risking empty stream
ms.Seek(0, SeekOrigin.Begin);
// Test and work with the stream here.
// If you need to start back at the beginning, be sure to Seek again.
}
finally
{
sw.Dispose();
}
}
As you can see, this code uses a StreamWriter to write the entire string (with proper encoding) out to the MemoryStream. This takes the hassle out of ensuring the entire byte array for the string is written.
Update: I stepped into issue with empty stream several time. It's enough to call Flush right after you've finished writing.

Try this "one-liner" from Delta's Blog, String To MemoryStream (C#).
MemoryStream stringInMemoryStream =
new MemoryStream(ASCIIEncoding.Default.GetBytes("Your string here"));
The string will be loaded into the MemoryStream, and you can read from it. See Encoding.GetBytes(...), which has also been implemented for a few other encodings.

You're using message.Length which returns the number of characters in the string, but you should be using the nubmer of bytes to read. You should use something like:
byte[] messageBytes = uniEncoding.GetBytes(message);
stringAsStream.Write(messageBytes, 0, messageBytes.Length);
You're then reading a single byte and expecting to get a character from it just by casting to char. UnicodeEncoding will use two bytes per character.
As Justin says you're also not seeking back to the beginning of the stream.
Basically I'm afraid pretty much everything is wrong here. Please give us the bigger picture and we can help you work out what you should really be doing. Using a StreamWriter to write and then a StreamReader to read is quite possibly what you want, but we can't really tell from just the brief bit of code you've shown.

I think it would be a lot more productive to use a TextWriter, in this case a StreamWriter to write to the MemoryStream. After that, as other have said, you need to "rewind" the MemoryStream using something like stringAsStream.Position = 0L;.
stringAsStream = new MemoryStream();
// create stream writer with UTF-16 (Unicode) encoding to write to the memory stream
using(StreamWriter sWriter = new StreamWriter(stringAsStream, UnicodeEncoding.Unicode))
{
sWriter.Write("Lorem ipsum.");
}
stringAsStream.Position = 0L; // rewind
Note that:
StreamWriter defaults to using an instance of UTF8Encoding unless specified otherwise. This instance of UTF8Encoding is constructed without a byte order mark (BOM)
Also, you don't have to create a new UnicodeEncoding() usually, since there's already one as a static member of the class for you to use in convenient utf-8, utf-16, and utf-32 flavors.
And then, finally (as others have said) you're trying to convert the bytes directly to chars, which they are not. If I had a memory stream and knew it was a string, I'd use a TextReader to get the string back from the bytes. It seems "dangerous" to me to mess around with the raw bytes.

You need to reset the stream to the beginning:
stringAsStream.Seek(0, SeekOrigin.Begin);
Console.WriteLine("Differs from:\t" + (char)stringAsStream.ReadByte());
This can also be done by setting the Position property to 0:
stringAsStream.Position = 0