My end goal is to use protobuf-net and GZipStream in an attempt to compress a List<MyCustomType> object to store in a varbinary(max) field in SQL Server. I'm working on unit tests to understand how everything works and fits together.
Target .NET framework is 3.5.
My current process is:
Serialize the data with protobuf-net (good).
Compress the serialized data from #1 with GZipStream (good).
Convert the compressed data to a base64 string (good).
At this point, the value from step #3 will be stored in a varbinary(max) field. I have no control over this. The steps resume with needing to take a base64 string and deserialize it to a concrete type.
Convert a base 64 string to a byte[] (good).
Decompress the data with GZipStream (good).
Deserialize the data with protobuf-net (bad).
Can someone assist with why the call to Serializer.Deserialize<string> returns null? I'm stuck on this one and hopefully a fresh set of eyes will help.
FWIW, I tried another version of this using List<T> where T is a custom class I created and I Deserialize<> still returns null.
FWIW 2, data.txt is a 4MB plaintext file residing on my C:.
[Test]
public void ForStackOverflow()
{
string data = "hi, my name is...";
//string data = File.ReadAllText(#"C:\Temp\data.txt");
string serializedBase64;
using (MemoryStream protobuf = new MemoryStream())
{
Serializer.Serialize(protobuf, data);
using (MemoryStream compressed = new MemoryStream())
{
using (GZipStream gzip = new GZipStream(compressed, CompressionMode.Compress))
{
byte[] s = protobuf.ToArray();
gzip.Write(s, 0, s.Length);
gzip.Close();
}
serializedBase64 = Convert.ToBase64String(compressed.ToArray());
}
}
byte[] base64byteArray = Convert.FromBase64String(serializedBase64);
using (MemoryStream base64Stream = new MemoryStream(base64byteArray))
{
using (GZipStream gzip = new GZipStream(base64Stream, CompressionMode.Decompress))
{
using (MemoryStream plainText = new MemoryStream())
{
byte[] buffer = new byte[4096];
int read;
while ((read = gzip.Read(buffer, 0, buffer.Length)) > 0)
{
plainText.Write(buffer, 0, read);
}
// why does this call to Deserialize return null?
string deserialized = Serializer.Deserialize<string>(plainText);
Assert.IsNotNull(deserialized);
Assert.AreEqual(data, deserialized);
}
}
}
}
Because you didn't rewind plainText after writing to it. Actually, that entire Stream is unnecessary - this works:
using (MemoryStream base64Stream = new MemoryStream(base64byteArray))
{
using (GZipStream gzip = new GZipStream(
base64Stream, CompressionMode.Decompress))
{
string deserialized = Serializer.Deserialize<string>(gzip);
Assert.IsNotNull(deserialized);
Assert.AreEqual(data, deserialized);
}
}
Likewise, this should work for the serialize:
using (MemoryStream compressed = new MemoryStream())
{
using (GZipStream gzip = new GZipStream(
compressed, CompressionMode.Compress, true))
{
Serializer.Serialize(gzip, data);
}
serializedBase64 = Convert.ToBase64String(
compressed.GetBuffer(), 0, (int)compressed.Length);
}
Related
I have a python zlib decompressor that takes default parameters as follows, where data is string:
import zlib
data_decompressed = zlib.decompress(data)
But, I don't know how I can compress a string in c# to be decompressed in python. I've tray the next piece of code but when I trie to decompresse 'incorrect header check' exception is trown.
static byte[] ZipContent(string entryName)
{
// remove whitespace from xml and convert to byte array
byte[] normalBytes;
using (StringWriter writer = new StringWriter())
{
//xml.Save(writer, SaveOptions.DisableFormatting);
System.Text.ASCIIEncoding encoding = new System.Text.ASCIIEncoding();
normalBytes = encoding.GetBytes(writer.ToString());
}
// zip into new, zipped, byte array
using (Stream memOutput = new MemoryStream())
using (ZipOutputStream zipOutput = new ZipOutputStream(memOutput))
{
zipOutput.SetLevel(6);
ZipEntry entry = new ZipEntry(entryName);
entry.CompressionMethod = CompressionMethod.Deflated;
entry.DateTime = DateTime.Now;
zipOutput.PutNextEntry(entry);
zipOutput.Write(normalBytes, 0, normalBytes.Length);
zipOutput.Finish();
byte[] newBytes = new byte[memOutput.Length];
memOutput.Seek(0, SeekOrigin.Begin);
memOutput.Read(newBytes, 0, newBytes.Length);
zipOutput.Close();
return newBytes;
}
}
Anyone could help me please?
Thank you.
UPDATE 1:
I've tried with defalte function as Shiraz Bhaiji has posted:
public static byte[] Deflate(byte[] data)
{
if (null == data || data.Length < 1) return null;
byte[] compressedBytes;
//write into a new memory stream wrapped by a deflate stream
using (MemoryStream ms = new MemoryStream())
{
using (DeflateStream deflateStream = new DeflateStream(ms, CompressionMode.Compress, true))
{
//write byte buffer into memorystream
deflateStream.Write(data, 0, data.Length);
deflateStream.Close();
//rewind memory stream and write to base 64 string
compressedBytes = new byte[ms.Length];
ms.Seek(0, SeekOrigin.Begin);
ms.Read(compressedBytes, 0, (int)ms.Length);
}
}
return compressedBytes;
}
The problem is that to work properly in python code I've to add the "-zlib.MAX_WBITS" argument to decompress as follows:
data_decompressed = zlib.decompress(data, -zlib.MAX_WBITS)
So, my new question is: is it possible to code a deflate method in C# which compression result could be decompressed with zlib.decompress(data) as defaults?
In C# the DeflateStream class supports zlib. See:
https://learn.microsoft.com/en-us/dotnet/api/system.io.compression.deflatestream?view=netframework-4.8
As you described with your edit, zlib.decompress(data, -zlib.MAX_WBITS) is the correct way to decompress data from C#'s DeflateStream. There are two formats at play here:
deflate - as in specification RFC 1951 - this is what's C# is producing
zlib - as in specification RFC 1950 - this is what's Python is expecting by default
What is the difference between the two? It's small, really:
zlib = [compression flag byte] + [flags byte] + deflate + [adler checksum]
(there are also optional dictionary bytes but we don't have to worry about them)
Therefore, to get zlib format from deflate, we need to prepend two bytes of flags, and append Adler-32 checksum. Luckily we have an answer on stackoverflow for the flags, see What does a zlib header look like? and implementing Adler-32 is not that hard. So suppose you have your MemoryStream ms, we would first write the two flag bytes
ms.Write(new byte[] {0x78,0x9c});
...then we would do exactly what's in your answer
using (DeflateStream deflateStream = new DeflateStream(ms, CompressionMode.Compress, true))
{
deflateStream.Write(data, 0, data.Length);
deflateStream.Close();
}
and, at last, compute the checksum and append it to the end of the stream:
uint a = 0;
uint b = 0;
for(int i = 0; i < data.Length; ++i)
{
a = (a + data[i]) % 65521;
b = (b + a) % 65521;
}
Sadly, I don't know a pretty way of writing uints into the stream. This is an ugly way:
ms.Write(new byte[] { (byte)(b>>8),
(byte)b,
(byte)(a>>8),
(byte)a
});
I have the following piece of code:
MemoryStream resultStream = new MemoryStream();
string users = ""//Really long string goes here
BinaryFormatter bFormatter = new BinaryFormatter();
using (MemoryStream assignedUsersStream = new MemoryStream())
{
bFormatter.Serialize(assignedUsersStream, users);
assignedUsersStream.Position = 0;
using (var compressionStream =
new DeflateStream(resultStream, CompressionLevel.Optimal))
{
assignedUsersStream.CopyTo(compressionStream);
Console.WriteLine("Compressed from {0} to {1} bytes.",
assignedUsersStream.Length.ToString(),
resultStream.Length.ToString());
}
}
the thing is that resultStream is always empty!
What am I doing wrong here?
Put your verification WriteLine outside of the using. The buffers haven't been flushed yet.
using (DeflateStream compressionStream = new DeflateStream(resultStream, CompressionLevel.Optimal))
{
assignedUsersStream.CopyTo(compressionStream);
//Console.WriteLine("Compressed from {0} to {1} bytes.",
// assignedUsersStream.Length.ToString(), resultStream.Length.ToString());
}
Console.WriteLine("Compressed from {0} to {1} bytes.",
assignedUsersStream.Length, resultStream.ToArray().Length);
And aside, you don't need all those ToString()s in a writeline.
PS: All a BinaryFormatter does with a string is write the bytes with length prefix. If you don't need the prefix (my guess), it could become:
string users = "";//Really long string goes here
byte[] result;
using (MemoryStream resultStream = new MemoryStream())
{
using (DeflateStream compressionStream = new DeflateStream(resultStream,
CompressionLevel.Optimal))
{
byte[] inBuffer = Encoding.UTF8.GetBytes(users);
compressionStream.Write(inBuffer, 0, inBuffer.Length);
}
result = resultStream.ToArray();
}
The reverse is just as easy but you'll need an estimate of the maximum length to create the read-buffer:
string users2 = null;
using (MemoryStream resultStream = new MemoryStream(result))
{
using (DeflateStream compressionStream = new DeflateStream(resultStream,
CompressionMode.Decompress))
{
byte[] outBuffer = new byte[2048]; // need an estimate here
int length = compressionStream.Read(outBuffer, 0, outBuffer.Length);
users2 = Encoding.UTF8.GetString(outBuffer, 0, length);
}
}
That is because the DeflateStream doesn't flush the data to the underlying stream until it is closed. After it is closed, resultStream will contain the compressed data. Note that by default, DeflateStream closes the underlying stream when it's closed, but you don't want that, so you need to pass true for the leaveOpen parameter. Also, you don't need 2 memory streams, you can just serialize directly to the compressionStream:
string users = ""; //Really long string goes here
BinaryFormatter bFormatter = new BinaryFormatter();
using (MemoryStream resultStream = new MemoryStream())
{
using (DeflateStream compressionStream = new DeflateStream(resultStream, CompressionLevel.Optimal, true))
{
bFormatter.Serialize(compressionStream, users);
Console.WriteLine(resultStream.Length); // 0 at this point
}
Console.WriteLine(resultStream.Length); // now contains the actual length
}
From the original answer (I don't have enough credits to vote down)
Put your control WriteLine outside of the using
This is incomplete and IMO therefore misleading. DeflateStream's Dispose(bool) implementation Closes the underlying resultStream when the DeflateStream is being Finalized after it's been Garbage Collected. When this happens, resultStream.Length will throw:
Unhandled Exception: System.ObjectDisposedException: Cannot access a closed Stream.
In other words, Thomas Levesque's note is critical: also set leaveOpen to true.
An interesting question with some good points raised by HH and TL.
I've come in late to this as I ran into this same problem and reading the conflicting answers. The initial response works! as does this test (over simplified to highlight the answer):
var inStream = new MemoryStream(data);
var outStream = new MemoryStream();
using (var compressor = new DeflateStream(outStream, CompressionLevel.Optimal))
{
inStream.CopyTo(compressor);
}
return outStream;
where the using block needs to complete, triggering the compressor's Dispose, which internally Flush()es so that outStream will be guaranteed to contain the complete compressed data.
I am trying to replicate the php function gzuncompress in C#
So far I got part of following code working. see comment and code below.
I thing the tricky bit is happening during byte[] and string convertion.
How can I fix this? and where did I missed??
I am using .Net 3.5 environment
var plaintext = Console.ReadLine();
Console.WriteLine("string to byte[] then to string");
byte[] buff = Encoding.UTF8.GetBytes(plaintext);
var compress = GZip.GZipCompress(buff);
//Uncompress working below
try
{
var unpressFromByte = GZip.GZipUncompress(compress);
Console.WriteLine("uncompress successful by uncompress byte[]");
}catch
{
Console.WriteLine("uncompress failed by uncompress byte[]");
}
var compressString = Encoding.UTF8.GetString(compress);
Console.WriteLine(compressString);
var compressBuff = Encoding.UTF8.GetBytes(compressString);
Console.WriteLine(Encoding.UTF8.GetString(compressBuff));
//Uncompress not working below by using string
//The magic number in GZip header is not correct
try
{
var uncompressFromString = GZip.GZipUncompress(compressBuff);
Console.WriteLine("uncompress successful by uncompress string");
}
catch
{
Console.WriteLine("uncompress failed by uncompress string");
}
code for class Gzip
public static class GZip
{
public static byte[] GZipUncompress(byte[] data)
{
using (var input = new MemoryStream(data))
using (var gzip = new GZipStream(input, CompressionMode.Decompress))
using (var output = new MemoryStream())
{
gzip.CopyTo(output);
return output.ToArray();
}
}
public static byte[] GZipCompress(byte[] data)
{
using (var input = new MemoryStream(data))
using (var output = new MemoryStream())
{
using (var gzip = new GZipStream(output, CompressionMode.Compress, true))
{
input.CopyTo(gzip);
}
return output.ToArray();
}
}
public static long CopyTo(this Stream source, Stream destination)
{
var buffer = new byte[2048];
int bytesRead;
long totalBytes = 0;
while ((bytesRead = source.Read(buffer, 0, buffer.Length)) > 0)
{
destination.Write(buffer, 0, bytesRead);
totalBytes += bytesRead;
}
return totalBytes;
}
}
This is inappropriate:
var compressString = Encoding.UTF8.GetString(compress);
compress isn't a UTF-8-encoded piece of text. You should treat it as arbitrary binary data - which isn't appropriate to pass into Encoding.GetString. If you really need to convert arbitrary binary data into text, use Convert.ToBase64String (and then reverse with Convert.FromBase64String):
var compressString = Convert.ToBase64String(compress);
Console.WriteLine(compressString);
var compressBuff = Convert.FromBase64String(compressString);
That may or may not match what PHP does, but it's a safe way of representing arbitrary binary data as text, unlike treating the binary data as if it were valid UTF-8-encoded text.
I am trying to replicate the php function gzuncompress in C#
Then use GZipStream or DeflateStream classes which are built into the .NET framework for this purpose.
I have been using the following code to Compress data in .Net 4.0:
public static byte[] CompressData(byte[] data_toCompress)
{
using (MemoryStream outFile = new MemoryStream())
{
using (MemoryStream inFile = new MemoryStream(data_toCompress))
using (GZipStream Compress = new GZipStream(outFile, CompressionMode.Compress))
{
inFile.CopyTo(Compress);
}
return outFile.ToArray();
}
}
However, in .Net 2.0 Stream.CopyTo method is not available. So, I tried making a replacement:
public static byte[] CompressData(byte[] data_toCompress)
{
using (MemoryStream outFile = new MemoryStream())
{
using (MemoryStream inFile = new MemoryStream(data_toCompress))
using (GZipStream Compress = new GZipStream(outFile, CompressionMode.Compress))
{
//inFile.CopyTo(Compress);
Compress.Write(inFile.GetBuffer(), (int)inFile.Position, (int)(inFile.Length - inFile.Position));
}
return outFile.ToArray();
}
}
The compression fails, though, when using the above attempt - I get an error saying:
MemoryStream's internal buffer cannot be accessed.
Could anyone offer any help on this issue? I'm really not sure what else to do here.
Thank you,
Evan
This is the code straight out of .Net 4.0 Stream.CopyTo method (bufferSize is 4096):
byte[] buffer = new byte[bufferSize];
int count;
while ((count = this.Read(buffer, 0, buffer.Length)) != 0)
destination.Write(buffer, 0, count);
Since you have access to the array already, why don't you do this:
using (MemoryStream outFile = new MemoryStream())
{
using (GZipStream Compress = new GZipStream(outFile, CompressionMode.Compress))
{
Compress.Write(data_toCompress, 0, data_toCompress.Length);
}
return outFile.ToArray();
}
Most likely in the sample code you are using inFile.GetBuffer() will throw an exception since you do not use the right constructor - not all MemoryStream instances allow you access to the internal buffer - you have to look for this in the documentation:
Initializes a new instance of the MemoryStream class based on the
specified region of a byte array, with the CanWrite property set as
specified, and the ability to call GetBuffer set as specified.
This should work - but is not needed anyway in the suggested solution:
using (MemoryStream inFile = new MemoryStream(data_toCompress,
0,
data_toCompress.Length,
false,
true))
Why are you constructing a memory stream with an array and then trying to pull the array back out of the memory stream?
You could just do Compress.Write(data_toCompress, 0, data_toCompress.Length);
If you need to replace the functionality of CopyTo, you can create a buffer array of some length, read data from the source stream and write that data to the destination stream.
You can try
infile.WriteTo(Compress);
try to replace the line:
Compress.Write(inFile.GetBuffer(), (int)inFile.Position, (int)(inFile.Length - inFile.Position));
with:
Compress.Write(data_toCompress, 0, data_toCompress.Length);
you can get rid of this line completely:
using (MemoryStream inFile = new MemoryStream(data_toCompress))
Edit: find an example here: Why does gzip/deflate compressing a small file result in many trailing zeroes?
You should manually read and write between these 2 streams:
private static void CopyStream(Stream from, Stream to)
{
int bufSize = 1024, count;
byte[] buffer = new byte[bufSize];
count = from.Read(buffer, 0, bufSize);
while (count > 0)
{
to.Write(buffer, 0, count);
count = from.Read(buffer, 0, bufSize);
}
}
The open-source NuGet package Stream.CopyTo implements Stream.CopyTo for all versions of the .NET Framework.
Available on GitHub and via NuGet (Install-Package Stream.CopyTo)
I am having an issue with this test function where I take an in memory string, compress it, and decompress it. The compression works great, but I can't seem to get the decompression to work.
//Compress
System.IO.MemoryStream outStream = new System.IO.MemoryStream();
GZipStream tinyStream = new GZipStream(outStream, CompressionMode.Compress);
mStream.Position = 0;
mStream.CopyTo(tinyStream);
//Decompress
outStream.Position = 0;
GZipStream bigStream = new GZipStream(outStream, CompressionMode.Decompress);
System.IO.MemoryStream bigStreamOut = new System.IO.MemoryStream();
bigStream.CopyTo(bigStreamOut);
//Results:
//bigStreamOut.Length == 0
//outStream.Position == the end of the stream.
I believe that bigStream out should at least have data in it, especially if my source stream (outStream) is being read. is this a MSFT bug or mine?
What happens in your code is that you keep opening streams, but you never close them.
In line 2, you create a GZipStream. This stream will not write anything to the underlying stream until it feels it’s the right time. You can tell it to by closing it.
However, if you close it, it will close the underlying stream (outStream) too. Therefore you can’t use mStream.Position = 0 on it.
You should always use using to ensure that all your streams get closed. Here is a variation on your code that works.
var inputString = "“ ... ”";
byte[] compressed;
string output;
using (var outStream = new MemoryStream())
{
using (var tinyStream = new GZipStream(outStream, CompressionMode.Compress))
using (var mStream = new MemoryStream(Encoding.UTF8.GetBytes(inputString)))
mStream.CopyTo(tinyStream);
compressed = outStream.ToArray();
}
// “compressed” now contains the compressed string.
// Also, all the streams are closed and the above is a self-contained operation.
using (var inStream = new MemoryStream(compressed))
using (var bigStream = new GZipStream(inStream, CompressionMode.Decompress))
using (var bigStreamOut = new MemoryStream())
{
bigStream.CopyTo(bigStreamOut);
output = Encoding.UTF8.GetString(bigStreamOut.ToArray());
}
// “output” now contains the uncompressed string.
Console.WriteLine(output);
This is a known issue: http://blogs.msdn.com/b/bclteam/archive/2006/05/10/592551.aspx
I have changed your code a bit so this one works:
var mStream = new MemoryStream(new byte[100]);
var outStream = new System.IO.MemoryStream();
using (var tinyStream = new GZipStream(outStream, CompressionMode.Compress))
{
mStream.CopyTo(tinyStream);
}
byte[] bb = outStream.ToArray();
//Decompress
var bigStream = new GZipStream(new MemoryStream(bb), CompressionMode.Decompress);
var bigStreamOut = new System.IO.MemoryStream();
bigStream.CopyTo(bigStreamOut);
The way to compress and decompress to and from a MemoryStream is:
public static Stream Compress(
Stream decompressed,
CompressionLevel compressionLevel = CompressionLevel.Fastest)
{
var compressed = new MemoryStream();
using (var zip = new GZipStream(compressed, compressionLevel, true))
{
decompressed.CopyTo(zip);
}
compressed.Seek(0, SeekOrigin.Begin);
return compressed;
}
public static Stream Decompress(Stream compressed)
{
var decompressed = new MemoryStream();
using (var zip = new GZipStream(compressed, CompressionMode.Decompress, true))
{
zip.CopyTo(decompressed);
}
decompressed.Seek(0, SeekOrigin.Begin);
return decompressed;
}
This leaves the compressed / decompressed stream open and as such usable after creating it.
Another implementation, in VB.NET:
Imports System.Runtime.CompilerServices
Imports System.IO
Imports System.IO.Compression
Public Module Compressor
<Extension()> _
Function CompressASCII(str As String) As Byte()
Dim bytes As Byte() = Encoding.ASCII.GetBytes(str)
Using ms As New MemoryStream
Using gzStream As New GZipStream(ms, CompressionMode.Compress)
gzStream.Write(bytes, 0, bytes.Length)
End Using
Return ms.ToArray
End Using
End Function
<Extension()> _
Function DecompressASCII(compressedString As Byte()) As String
Using ms As New MemoryStream(compressedString)
Using gzStream As New GZipStream(ms, CompressionMode.Decompress)
Using sr As New StreamReader(gzStream, Encoding.ASCII)
Return sr.ReadToEnd
End Using
End Using
End Using
End Function
Sub TestCompression()
Dim input As String = "fh3o047gh"
Dim compressed As Byte() = input.CompressASCII()
Dim decompressed As String = compressed.DecompressASCII()
If input <> decompressed Then
Throw New ApplicationException("failure!")
End If
End Sub
End Module
public static byte[] compress(byte[] data)
{
using (MemoryStream outStream = new MemoryStream())
{
using (GZipStream gzipStream = new GZipStream(outStream, CompressionMode.Compress))
using (MemoryStream srcStream = new MemoryStream(data))
srcStream.CopyTo(gzipStream);
return outStream.ToArray();
}
}
public static byte[] decompress(byte[] compressed)
{
using (MemoryStream inStream = new MemoryStream(compressed))
using (GZipStream gzipStream = new GZipStream(inStream, CompressionMode.Decompress))
using (MemoryStream outStream = new MemoryStream())
{
gzipStream.CopyTo(outStream);
return outStream.ToArray();
}
}
If you are attempting to use the MemoryStream (e.g. passing it into another function) but receiving the Exception "Cannot access a closed Stream." then there is another GZipStream constructor you can use that will help you.
By passing in a true to the leaveOpen parameter, you can instruct GZipStream to leave the stream open after disposing of itself, by default it closes the target stream (which I didn't expect). https://msdn.microsoft.com/en-us/library/27ck2z1y(v=vs.110).aspx
using (FileStream fs = File.OpenRead(f))
using (var compressed = new MemoryStream())
{
//Instruct GZipStream to leave the stream open after performing the compression.
using (var gzipstream = new GZipStream(compressed, CompressionLevel.Optimal, true))
fs.CopyTo(gzipstream);
//Do something with the memorystream
compressed.Seek(0, SeekOrigin.Begin);
MyFunction(compressed);
}
I had an issue where *.CopyTo(stream)* would end up with a byte[0] result.
The solution was to add .Position=0 before calling .CopyTo(stream)
Answered here
I also use a BinaryFormatter that would throw an 'End of stream encountered before parsing was completed' exception if position was not set to 0 before deserialization.
Answered here
This is the code that worked for me.
public static byte[] SerializeAndCompressStateInformation(this IPluginWithStateInfo plugin, Dictionary<string, object> stateInfo)
{
byte[] retArr = new byte[] { byte.MinValue };
try
{
using (MemoryStream msCompressed = new MemoryStream())//what gzip writes to
{
using (GZipStream gZipStream = new GZipStream(msCompressed, CompressionMode.Compress))//setting up gzip
using (MemoryStream msToCompress = new MemoryStream())//what the settings will serialize to
{
BinaryFormatter formatter = new BinaryFormatter();
//serialize the info into bytes
formatter.Serialize(msToCompress, stateInfo);
//reset to 0 to read from beginning byte[0] fix.
msToCompress.Position = 0;
//this then does the compression
msToCompress.CopyTo(gZipStream);
}
//the compressed data as an array of bytes
retArr = msCompressed.ToArray();
}
}
catch (Exception ex)
{
Logger.Error(ex.Message, ex);
throw ex;
}
return retArr;
}
public static Dictionary<string, object> DeserializeAndDecompressStateInformation(this IPluginWithStateInfo plugin, byte[] stateInfo)
{
Dictionary<string, object> settings = new Dictionary<string, object>();
try
{
using (MemoryStream msDecompressed = new MemoryStream()) //the stream that will hold the decompressed data
{
using (MemoryStream msCompressed = new MemoryStream(stateInfo))//the compressed data
using (GZipStream gzDecomp = new GZipStream(msCompressed, CompressionMode.Decompress))//the gzip that will decompress
{
msCompressed.Position = 0;//fix for byte[0]
gzDecomp.CopyTo(msDecompressed);//decompress the data
}
BinaryFormatter formatter = new BinaryFormatter();
//prevents 'End of stream encountered' error
msDecompressed.Position = 0;
//change the decompressed data to the object
settings = formatter.Deserialize(msDecompressed) as Dictionary<string, object>;
}
}
catch (Exception ex)
{
Logger.Error(ex.Message, ex);
throw ex;
}
return settings;
}
I thought I would share this answer for anyone interested in reproducing this on PowerShell, the code is mostly inspired from Timwi's helpful answer, however unfortunately as of now there is no implementation for the using statement like on C# for PowerShell, hence the need to manually dispose the streams before output.
Functions below requires PowerShell 5.0+.
Required using statements and ArgumentCompleter for -Encoding Parameter.
Improved versions of these 2 functions as well as Compression From File Path and Expansion from File Path can be found in this repo as well as in the PowerShell Gallery.
using namespace System.Text
using namespace System.IO
using namespace System.IO.Compression
using namespace System.Collections
using namespace System.Management.Automation
using namespace System.Collections.Generic
using namespace System.Management.Automation.Language
Add-Type -AssemblyName System.IO.Compression
class EncodingCompleter : IArgumentCompleter {
[IEnumerable[CompletionResult]] CompleteArgument (
[string] $commandName,
[string] $parameterName,
[string] $wordToComplete,
[CommandAst] $commandAst,
[IDictionary] $fakeBoundParameters
) {
[CompletionResult[]] $arguments = foreach($enc in [Encoding]::GetEncodings().Name) {
if($enc.StartsWith($wordToComplete)) {
[CompletionResult]::new($enc)
}
}
return $arguments
}
}
Compression from string to Base64 GZip compressed string:
function Compress-GzipString {
[cmdletbinding()]
param(
[Parameter(Mandatory, ValueFromPipeline)]
[string] $String,
[Parameter()]
[ArgumentCompleter([EncodingCompleter])]
[string] $Encoding = 'utf-8',
[Parameter()]
[CompressionLevel] $CompressionLevel = 'Optimal'
)
try {
$enc = [Encoding]::GetEncoding($Encoding)
$outStream = [MemoryStream]::new()
$gzip = [GZipStream]::new($outStream, [CompressionMode]::Compress, $CompressionLevel)
$inStream = [MemoryStream]::new($enc.GetBytes($string))
$inStream.CopyTo($gzip)
}
catch {
$PSCmdlet.WriteError($_)
}
finally {
$gzip, $outStream, $inStream | ForEach-Object Dispose
}
try {
[Convert]::ToBase64String($outStream.ToArray())
}
catch {
$PSCmdlet.WriteError($_)
}
}
Expansion from Base64 GZip compressed string to string:
function Expand-GzipString {
[cmdletbinding()]
param(
[Parameter(Mandatory, ValueFromPipeline)]
[string] $String,
[Parameter()]
[ArgumentCompleter([EncodingCompleter])]
[string] $Encoding = 'utf-8'
)
try {
$enc = [Encoding]::GetEncoding($Encoding)
$bytes = [Convert]::FromBase64String($String)
$outStream = [MemoryStream]::new()
$inStream = [MemoryStream]::new($bytes)
$gzip = [GZipStream]::new($inStream, [CompressionMode]::Decompress)
$gzip.CopyTo($outStream)
$enc.GetString($outStream.ToArray())
}
catch {
$PSCmdlet.WriteError($_)
}
finally {
$gzip, $outStream, $inStream | ForEach-Object Dispose
}
}
And for the little Length comparison, querying the Loripsum API:
$loremIp = Invoke-RestMethod loripsum.net/api/10/long
$compressedLoremIp = Compress-GzipString $loremIp
$loremIp, $compressedLoremIp | Select-Object Length
Length
------
8353
4940
(Expand-GzipString $compressedLoremIp) -eq $loremIp # => Should be True
If you still need it, you can use GZipStream constructor wit boolean argument (there are two such constructors) and pass true value there:
tinyStream = new GZipStream(outStream, CompressionMode.Compress, true);
In that case, when you close your tynyStream, your out stream will be still opened. Don't forget to copy data:
mStream.CopyTo(tinyStream);
tinyStream.Close();
Now you've got memory stream outStream with zipped data
Bugs and kisses for U
Good luck
Please refer to below link, It is avoid to use double MemoryStream.
https://stackoverflow.com/a/53644256/1979406