We Keep Coding

Encrypt and compress FileStream and save to same file

I have a method to compress and encrypt a stream.
private static void CompressThenEncrypt(string inputFileName, ICryptoTransform encryptor)
{
using (var inputFileStream = new FileStream(inputFileName, FileMode.Open, FileAccess.ReadWrite))
using (var cryptoStream = new CryptoStream(inputFileStream, encryptor, CryptoStreamMode.Write))
using (var zipStream = new GZipStream(cryptoStream, CompressionMode.Compress))
{
inputFileStream.CopyTo(zipStream);
}
}
Which does work, but it appends compressed and encrypted data to raw one I have in that file.
So if my file contains:
kkk
Then after compression and encryption it does look for example like that:
kkkㆆ鬁⠕⟶ꏙᇚ셑襜㷡ꕢ束৘㺝娥☪
Do anyone have any idea how to overwrite new data?

I suggest:
Write to a new file
Delete the old file
Rename the new file to the old file.

Write to a temporary MemoryStream. Something like:
private static void CompressThenEncrypt(string inputFileName, ICryptoTransform encryptor)
{
using (var fileStream = new FileStream(inputFileName,FileMode.Open, FileAccess.ReadWrite))
using (var memoryStream = new MemoryStream())
using (var cryptoStream = new CryptoStream(memoryStream, encryptor, CryptoStreamMode.Write))
using (var zipStream = new GZipStream(cryptoStream, CompressionMode.Compress))
{
fileStream.CopyTo(zipStream);
cryptoStream.FlushFinalBlock();
fileStream.SetLength(0);
memoryStream.Position = 0;
memoryStream.CopyTo(fileStream);
}
}
I think that should work, from the top of my head. Not sure you would have to set the length, and I'm not sure you can get away with not resetting the position of the MemoryStream. I haven't tested this code. If you are concerned about memory usage, you can always buffer the usage, but that would require more complicated code.

DeflateStream doesnt work on MemoryStream?

I have the following piece of code:
MemoryStream resultStream = new MemoryStream();
string users = ""//Really long string goes here
BinaryFormatter bFormatter = new BinaryFormatter();
using (MemoryStream assignedUsersStream = new MemoryStream())
{
bFormatter.Serialize(assignedUsersStream, users);
assignedUsersStream.Position = 0;
using (var compressionStream =
new DeflateStream(resultStream, CompressionLevel.Optimal))
{
assignedUsersStream.CopyTo(compressionStream);
Console.WriteLine("Compressed from {0} to {1} bytes.",
assignedUsersStream.Length.ToString(),
resultStream.Length.ToString());
}
}
the thing is that resultStream is always empty!
What am I doing wrong here?

Put your verification WriteLine outside of the using. The buffers haven't been flushed yet.
using (DeflateStream compressionStream = new DeflateStream(resultStream, CompressionLevel.Optimal))
{
assignedUsersStream.CopyTo(compressionStream);
//Console.WriteLine("Compressed from {0} to {1} bytes.",
// assignedUsersStream.Length.ToString(), resultStream.Length.ToString());
}
Console.WriteLine("Compressed from {0} to {1} bytes.",
assignedUsersStream.Length, resultStream.ToArray().Length);
And aside, you don't need all those ToString()s in a writeline.
PS: All a BinaryFormatter does with a string is write the bytes with length prefix. If you don't need the prefix (my guess), it could become:
string users = "";//Really long string goes here
byte[] result;
using (MemoryStream resultStream = new MemoryStream())
{
using (DeflateStream compressionStream = new DeflateStream(resultStream,
CompressionLevel.Optimal))
{
byte[] inBuffer = Encoding.UTF8.GetBytes(users);
compressionStream.Write(inBuffer, 0, inBuffer.Length);
}
result = resultStream.ToArray();
}
The reverse is just as easy but you'll need an estimate of the maximum length to create the read-buffer:
string users2 = null;
using (MemoryStream resultStream = new MemoryStream(result))
{
using (DeflateStream compressionStream = new DeflateStream(resultStream,
CompressionMode.Decompress))
{
byte[] outBuffer = new byte[2048]; // need an estimate here
int length = compressionStream.Read(outBuffer, 0, outBuffer.Length);
users2 = Encoding.UTF8.GetString(outBuffer, 0, length);
}
}

That is because the DeflateStream doesn't flush the data to the underlying stream until it is closed. After it is closed, resultStream will contain the compressed data. Note that by default, DeflateStream closes the underlying stream when it's closed, but you don't want that, so you need to pass true for the leaveOpen parameter. Also, you don't need 2 memory streams, you can just serialize directly to the compressionStream:
string users = ""; //Really long string goes here
BinaryFormatter bFormatter = new BinaryFormatter();
using (MemoryStream resultStream = new MemoryStream())
{
using (DeflateStream compressionStream = new DeflateStream(resultStream, CompressionLevel.Optimal, true))
{
bFormatter.Serialize(compressionStream, users);
Console.WriteLine(resultStream.Length); // 0 at this point
}
Console.WriteLine(resultStream.Length); // now contains the actual length
}

From the original answer (I don't have enough credits to vote down)
Put your control WriteLine outside of the using
This is incomplete and IMO therefore misleading. DeflateStream's Dispose(bool) implementation Closes the underlying resultStream when the DeflateStream is being Finalized after it's been Garbage Collected. When this happens, resultStream.Length will throw:
Unhandled Exception: System.ObjectDisposedException: Cannot access a closed Stream.
In other words, Thomas Levesque's note is critical: also set leaveOpen to true.
An interesting question with some good points raised by HH and TL.

I've come in late to this as I ran into this same problem and reading the conflicting answers. The initial response works! as does this test (over simplified to highlight the answer):
var inStream = new MemoryStream(data);
var outStream = new MemoryStream();
using (var compressor = new DeflateStream(outStream, CompressionLevel.Optimal))
{
inStream.CopyTo(compressor);
}
return outStream;
where the using block needs to complete, triggering the compressor's Dispose, which internally Flush()es so that outStream will be guaranteed to contain the complete compressed data.

Crypto stream: read data error

I got this (i also tried crStream.CopyTo(ms)):
var cryptic = new DESCryptoServiceProvider();
cryptic.Key = ASCIIEncoding.ASCII.GetBytes(passKey);
cryptic.IV = ASCIIEncoding.ASCII.GetBytes(passKey);
Stream crStream = new CryptoStream(data, cryptic.CreateEncryptor(), CryptoStreamMode.Write);
Stream ms = new MemoryStream();
var buffer = new byte[0x10000];
int n;
while ((n = crStream.Read(buffer, 0, buffer.Length)) != 0) // Exception occurs here
ms.Write(buffer, 0, n);
crStream.Close();
Data = Stream and contains a binary serialized class
The following exception occurs when i run it:
"Stream does not support reading."
What i am trying to accomplish is simply encrypt data from a stream. So i have an incoming stream and i want to encrypt that data and put it into the memory stream. This will then be compressed and saved to a file.

the error says everything: you create the stream for encryption (= put plain-text into and get encrypted output, in write):
Stream crStream = new CryptoStream(data, cryptic.CreateEncryptor(), CryptoStreamMode.Write);
Just have a look at the MSDN-Documentation for CryptoStream - there is a example included of how to do it right - it's basically this part (right from MSDN):
using (MemoryStream msEncrypt = new MemoryStream())
using (CryptoStream csEncrypt = new CryptoStream(msEncrypt, encryptor, CryptoStreamMode.Write))
{
using (StreamWriter swEncrypt = new StreamWriter(csEncrypt))
{
//Write all data to the stream.
swEncrypt.Write(plainText);
}
encrypted = msEncrypt.ToArray();
}

Error about padding occurs during my attempt to decrypt string data from a FileStream using Rijndael

I'm trying to encrypt some strings (actually a walkaround for encrypt an XElement object) into a file using Rijndael. However when decrypting, a CryptographicException will be thrown with the message "Padding is invalid and cannot be removed".
I've been searching several posts for possible solutions, in which someone mentions the PaddingMode, the FlushFinalBlock() method, and others.
Also someone has mentioned that it's because reading directly from a FileStream will set the length of the stream to that of the file, thus leaving no room for padding, but I don't know how to resolve the issue. Please kindly give your suggestions.
var root = new XElement("Users",
new XElement("User", new XAttribute("id", "1"), "User1"),
new XElement("User", new XAttribute("id", "2"), "User2"));
var r = Rijndael.Create();
r.Padding = PaddingMode.PKCS7;
using (var fs = File.Open(#"D:\user.xml", FileMode.Create))
using (var cs = new CryptoStream(fs, r.CreateEncryptor(r.Key, r.IV), CryptoStreamMode.Write))
using (var sw = new StreamWriter(cs, Encoding.Unicode))
{
sw.Write(root.ToString());
cs.FlushFinalBlock();
}
var r = Rijndael.Create();
r.Padding = PaddingMode.PKCS7;
using(var fs = File.Open(#"D:\user.xml", FileMode.Open, FileAccess.ReadWrite))
using(var cs = new CryptoStream(fs, r.CreateDecryptor(r.Key, r.IV), CryptoStreamMode.Read))
using (var sr = new StreamReader(cs, Encoding.Unicode))
{
var root = sr.ReadToEnd();
}

I believe you need to flush the stream writer as well.
As a note, make sure you actually use a random IV ( which you are doing here). Sometimes people write code like this and then when the go to use it to decrypt the file later, the realize they don't have the IV and so they just hard code it. Just prepend it to the file.
{
sw.Write(root.ToString());
cs.FlushFinalBlock();
sw.flush();
}

How do I use GZipStream with System.IO.MemoryStream?

I am having an issue with this test function where I take an in memory string, compress it, and decompress it. The compression works great, but I can't seem to get the decompression to work.
//Compress
System.IO.MemoryStream outStream = new System.IO.MemoryStream();
GZipStream tinyStream = new GZipStream(outStream, CompressionMode.Compress);
mStream.Position = 0;
mStream.CopyTo(tinyStream);
//Decompress
outStream.Position = 0;
GZipStream bigStream = new GZipStream(outStream, CompressionMode.Decompress);
System.IO.MemoryStream bigStreamOut = new System.IO.MemoryStream();
bigStream.CopyTo(bigStreamOut);
//Results:
//bigStreamOut.Length == 0
//outStream.Position == the end of the stream.
I believe that bigStream out should at least have data in it, especially if my source stream (outStream) is being read. is this a MSFT bug or mine?

What happens in your code is that you keep opening streams, but you never close them.
In line 2, you create a GZipStream. This stream will not write anything to the underlying stream until it feels it’s the right time. You can tell it to by closing it.
However, if you close it, it will close the underlying stream (outStream) too. Therefore you can’t use mStream.Position = 0 on it.
You should always use using to ensure that all your streams get closed. Here is a variation on your code that works.
var inputString = "“ ... ”";
byte[] compressed;
string output;
using (var outStream = new MemoryStream())
{
using (var tinyStream = new GZipStream(outStream, CompressionMode.Compress))
using (var mStream = new MemoryStream(Encoding.UTF8.GetBytes(inputString)))
mStream.CopyTo(tinyStream);
compressed = outStream.ToArray();
}
// “compressed” now contains the compressed string.
// Also, all the streams are closed and the above is a self-contained operation.
using (var inStream = new MemoryStream(compressed))
using (var bigStream = new GZipStream(inStream, CompressionMode.Decompress))
using (var bigStreamOut = new MemoryStream())
{
bigStream.CopyTo(bigStreamOut);
output = Encoding.UTF8.GetString(bigStreamOut.ToArray());
}
// “output” now contains the uncompressed string.
Console.WriteLine(output);

This is a known issue: http://blogs.msdn.com/b/bclteam/archive/2006/05/10/592551.aspx
I have changed your code a bit so this one works:
var mStream = new MemoryStream(new byte[100]);
var outStream = new System.IO.MemoryStream();
using (var tinyStream = new GZipStream(outStream, CompressionMode.Compress))
{
mStream.CopyTo(tinyStream);
}
byte[] bb = outStream.ToArray();
//Decompress
var bigStream = new GZipStream(new MemoryStream(bb), CompressionMode.Decompress);
var bigStreamOut = new System.IO.MemoryStream();
bigStream.CopyTo(bigStreamOut);

The way to compress and decompress to and from a MemoryStream is:
public static Stream Compress(
Stream decompressed,
CompressionLevel compressionLevel = CompressionLevel.Fastest)
{
var compressed = new MemoryStream();
using (var zip = new GZipStream(compressed, compressionLevel, true))
{
decompressed.CopyTo(zip);
}
compressed.Seek(0, SeekOrigin.Begin);
return compressed;
}
public static Stream Decompress(Stream compressed)
{
var decompressed = new MemoryStream();
using (var zip = new GZipStream(compressed, CompressionMode.Decompress, true))
{
zip.CopyTo(decompressed);
}
decompressed.Seek(0, SeekOrigin.Begin);
return decompressed;
}
This leaves the compressed / decompressed stream open and as such usable after creating it.

Another implementation, in VB.NET:
Imports System.Runtime.CompilerServices
Imports System.IO
Imports System.IO.Compression
Public Module Compressor
<Extension()> _
Function CompressASCII(str As String) As Byte()
Dim bytes As Byte() = Encoding.ASCII.GetBytes(str)
Using ms As New MemoryStream
Using gzStream As New GZipStream(ms, CompressionMode.Compress)
gzStream.Write(bytes, 0, bytes.Length)
End Using
Return ms.ToArray
End Using
End Function
<Extension()> _
Function DecompressASCII(compressedString As Byte()) As String
Using ms As New MemoryStream(compressedString)
Using gzStream As New GZipStream(ms, CompressionMode.Decompress)
Using sr As New StreamReader(gzStream, Encoding.ASCII)
Return sr.ReadToEnd
End Using
End Using
End Using
End Function
Sub TestCompression()
Dim input As String = "fh3o047gh"
Dim compressed As Byte() = input.CompressASCII()
Dim decompressed As String = compressed.DecompressASCII()
If input <> decompressed Then
Throw New ApplicationException("failure!")
End If
End Sub
End Module

public static byte[] compress(byte[] data)
{
using (MemoryStream outStream = new MemoryStream())
{
using (GZipStream gzipStream = new GZipStream(outStream, CompressionMode.Compress))
using (MemoryStream srcStream = new MemoryStream(data))
srcStream.CopyTo(gzipStream);
return outStream.ToArray();
}
}
public static byte[] decompress(byte[] compressed)
{
using (MemoryStream inStream = new MemoryStream(compressed))
using (GZipStream gzipStream = new GZipStream(inStream, CompressionMode.Decompress))
using (MemoryStream outStream = new MemoryStream())
{
gzipStream.CopyTo(outStream);
return outStream.ToArray();
}
}

If you are attempting to use the MemoryStream (e.g. passing it into another function) but receiving the Exception "Cannot access a closed Stream." then there is another GZipStream constructor you can use that will help you.
By passing in a true to the leaveOpen parameter, you can instruct GZipStream to leave the stream open after disposing of itself, by default it closes the target stream (which I didn't expect). https://msdn.microsoft.com/en-us/library/27ck2z1y(v=vs.110).aspx
using (FileStream fs = File.OpenRead(f))
using (var compressed = new MemoryStream())
{
//Instruct GZipStream to leave the stream open after performing the compression.
using (var gzipstream = new GZipStream(compressed, CompressionLevel.Optimal, true))
fs.CopyTo(gzipstream);
//Do something with the memorystream
compressed.Seek(0, SeekOrigin.Begin);
MyFunction(compressed);
}

I had an issue where *.CopyTo(stream)* would end up with a byte[0] result.
The solution was to add .Position=0 before calling .CopyTo(stream)
Answered here
I also use a BinaryFormatter that would throw an 'End of stream encountered before parsing was completed' exception if position was not set to 0 before deserialization.
Answered here
This is the code that worked for me.
public static byte[] SerializeAndCompressStateInformation(this IPluginWithStateInfo plugin, Dictionary<string, object> stateInfo)
{
byte[] retArr = new byte[] { byte.MinValue };
try
{
using (MemoryStream msCompressed = new MemoryStream())//what gzip writes to
{
using (GZipStream gZipStream = new GZipStream(msCompressed, CompressionMode.Compress))//setting up gzip
using (MemoryStream msToCompress = new MemoryStream())//what the settings will serialize to
{
BinaryFormatter formatter = new BinaryFormatter();
//serialize the info into bytes
formatter.Serialize(msToCompress, stateInfo);
//reset to 0 to read from beginning byte[0] fix.
msToCompress.Position = 0;
//this then does the compression
msToCompress.CopyTo(gZipStream);
}
//the compressed data as an array of bytes
retArr = msCompressed.ToArray();
}
}
catch (Exception ex)
{
Logger.Error(ex.Message, ex);
throw ex;
}
return retArr;
}
public static Dictionary<string, object> DeserializeAndDecompressStateInformation(this IPluginWithStateInfo plugin, byte[] stateInfo)
{
Dictionary<string, object> settings = new Dictionary<string, object>();
try
{
using (MemoryStream msDecompressed = new MemoryStream()) //the stream that will hold the decompressed data
{
using (MemoryStream msCompressed = new MemoryStream(stateInfo))//the compressed data
using (GZipStream gzDecomp = new GZipStream(msCompressed, CompressionMode.Decompress))//the gzip that will decompress
{
msCompressed.Position = 0;//fix for byte[0]
gzDecomp.CopyTo(msDecompressed);//decompress the data
}
BinaryFormatter formatter = new BinaryFormatter();
//prevents 'End of stream encountered' error
msDecompressed.Position = 0;
//change the decompressed data to the object
settings = formatter.Deserialize(msDecompressed) as Dictionary<string, object>;
}
}
catch (Exception ex)
{
Logger.Error(ex.Message, ex);
throw ex;
}
return settings;
}

I thought I would share this answer for anyone interested in reproducing this on PowerShell, the code is mostly inspired from Timwi's helpful answer, however unfortunately as of now there is no implementation for the using statement like on C# for PowerShell, hence the need to manually dispose the streams before output.
Functions below requires PowerShell 5.0+.
Required using statements and ArgumentCompleter for -Encoding Parameter.
Improved versions of these 2 functions as well as Compression From File Path and Expansion from File Path can be found in this repo as well as in the PowerShell Gallery.
using namespace System.Text
using namespace System.IO
using namespace System.IO.Compression
using namespace System.Collections
using namespace System.Management.Automation
using namespace System.Collections.Generic
using namespace System.Management.Automation.Language
Add-Type -AssemblyName System.IO.Compression
class EncodingCompleter : IArgumentCompleter {
[IEnumerable[CompletionResult]] CompleteArgument (
[string] $commandName,
[string] $parameterName,
[string] $wordToComplete,
[CommandAst] $commandAst,
[IDictionary] $fakeBoundParameters
) {
[CompletionResult[]] $arguments = foreach($enc in [Encoding]::GetEncodings().Name) {
if($enc.StartsWith($wordToComplete)) {
[CompletionResult]::new($enc)
}
}
return $arguments
}
}
Compression from string to Base64 GZip compressed string:
function Compress-GzipString {
[cmdletbinding()]
param(
[Parameter(Mandatory, ValueFromPipeline)]
[string] $String,
[Parameter()]
[ArgumentCompleter([EncodingCompleter])]
[string] $Encoding = 'utf-8',
[Parameter()]
[CompressionLevel] $CompressionLevel = 'Optimal'
)
try {
$enc = [Encoding]::GetEncoding($Encoding)
$outStream = [MemoryStream]::new()
$gzip = [GZipStream]::new($outStream, [CompressionMode]::Compress, $CompressionLevel)
$inStream = [MemoryStream]::new($enc.GetBytes($string))
$inStream.CopyTo($gzip)
}
catch {
$PSCmdlet.WriteError($_)
}
finally {
$gzip, $outStream, $inStream | ForEach-Object Dispose
}
try {
[Convert]::ToBase64String($outStream.ToArray())
}
catch {
$PSCmdlet.WriteError($_)
}
}
Expansion from Base64 GZip compressed string to string:
function Expand-GzipString {
[cmdletbinding()]
param(
[Parameter(Mandatory, ValueFromPipeline)]
[string] $String,
[Parameter()]
[ArgumentCompleter([EncodingCompleter])]
[string] $Encoding = 'utf-8'
)
try {
$enc = [Encoding]::GetEncoding($Encoding)
$bytes = [Convert]::FromBase64String($String)
$outStream = [MemoryStream]::new()
$inStream = [MemoryStream]::new($bytes)
$gzip = [GZipStream]::new($inStream, [CompressionMode]::Decompress)
$gzip.CopyTo($outStream)
$enc.GetString($outStream.ToArray())
}
catch {
$PSCmdlet.WriteError($_)
}
finally {
$gzip, $outStream, $inStream | ForEach-Object Dispose
}
}
And for the little Length comparison, querying the Loripsum API:
$loremIp = Invoke-RestMethod loripsum.net/api/10/long
$compressedLoremIp = Compress-GzipString $loremIp
$loremIp, $compressedLoremIp | Select-Object Length
Length
------
8353
4940
(Expand-GzipString $compressedLoremIp) -eq $loremIp # => Should be True

If you still need it, you can use GZipStream constructor wit boolean argument (there are two such constructors) and pass true value there:
tinyStream = new GZipStream(outStream, CompressionMode.Compress, true);
In that case, when you close your tynyStream, your out stream will be still opened. Don't forget to copy data:
mStream.CopyTo(tinyStream);
tinyStream.Close();
Now you've got memory stream outStream with zipped data
Bugs and kisses for U
Good luck

Please refer to below link, It is avoid to use double MemoryStream.
https://stackoverflow.com/a/53644256/1979406