We Keep Coding

Unable to resolve compression, encryption, decryption, decompression stream implementation

I've been fighting with chained using statements, and am unable to resolve the latest in a long line of implementation issues. I need to compress, then encrypt and append the generated IV to the selected file. This all appears to work correctly, however i'm unable to unwind the process. After looking at several similar stack postings and articles i'm still unable to get it to work and am now after more direct assistance.
The latest thrown error is System.IO.InvalidDataException: 'Found invalid data while decoding.' It appears that the decryption stream isn't functioning as intended and that's throwing the decompression stream out of wack.
byte[] key;
byte[] salt;
const int keySize = 256;
const int blockSize = keySize;
byte[] iv = new byte[blockSize / 8];//size to bits
RijndaelManaged rjndl;
RNGCryptoServiceProvider cRng;
void InitializeCryptor() {
//Temporarily define the salt & key
salt = Encoding.UTF8.GetBytes("SaltShouldBeAtLeast8Bytes");
key = new Rfc2898DeriveBytes("MyL0ngPa$$phra$e", salt, 4).GetBytes(keySize / 8);
//Initialize the crypto RNG generator
cRng = new RNGCryptoServiceProvider();
// Create instance of Rijndael (AES) for symetric encryption of the data.
rjndl = new RijndaelManaged();
rjndl.KeySize = keySize;
rjndl.BlockSize = blockSize;
rjndl.Mode = CipherMode.CBC;
}
void CompressAndEncryptFile(string relativeFilePath, string fileName) {
//Create a unique IV each time
cRng.GetBytes(iv);
//Create encryptor
rjndl.Key = key;
rjndl.IV = iv;
ICryptoTransform encryptor = rjndl.CreateEncryptor(rjndl.Key, rjndl.IV);
//Create file specific output sub-directory
Directory.CreateDirectory(Path.Combine(outputPath, relativeFilePath));
//Read and compress file into memory stream
using (FileStream readStream = File.OpenRead(Path.Combine(initialpath, relativeFilePath, fileName)))
using (FileStream writeStream = new FileStream(Path.Combine(outputPath, relativeFilePath, fileName + ".dat"), FileMode.Create))
using (CryptoStream encryptStream = new CryptoStream(writeStream, encryptor, CryptoStreamMode.Write))
using (DeflateStream compStream = new DeflateStream(encryptStream, CompressionLevel.Optimal)) {
//Write the following to the FileStream for the encrypted file:
// - length of the IV
// - the IV
byte[] ivSize = BitConverter.GetBytes(rjndl.IV.Length);
writeStream.Write(ivSize, 0, 4);
writeStream.Write(rjndl.IV, 0, rjndl.BlockSize / 8);
readStream.CopyTo(compStream);
}
}
void DecryptAndDecompressFile(string relativeFilePath) {
string outputPath = Path.Combine(initialpath, "Unpack");
Directory.CreateDirectory(outputPath);
using (FileStream readStream = new FileStream(Path.Combine(initialpath, manifestData.version, relativeFilePath + ".dat"), FileMode.Open)) {
byte[] tmpLength = new byte[4];
//Read length of IV
readStream.Seek(0, SeekOrigin.Begin);
readStream.Read(tmpLength, 0, 3);
int ivLength = BitConverter.ToInt32(tmpLength, 0);
byte[] readIv = new byte[ivLength];
//Read IV
readStream.Seek(4, SeekOrigin.Begin);
readStream.Read(readIv, 0, ivLength);
rjndl.IV = readIv;
//Start at beginning of encrypted data
readStream.Seek(4 + ivLength, SeekOrigin.Begin);
//Create decryptor
ICryptoTransform decryptor = rjndl.CreateEncryptor(key, readIv);
using (CryptoStream decryptStream = new CryptoStream(readStream, decryptor, CryptoStreamMode.Read))
using (DeflateStream decompStream = new DeflateStream(decryptStream, CompressionMode.Decompress))
using (FileStream writeStream = new FileStream(Path.Combine(outputPath, relativeFilePath), FileMode.Create)) {
decompStream.CopyTo(writeStream);
}
}
}
For those who like to point to other similar stack questions and vote to close/duplicate without offering support, the following are the threads, and posts I've worked through first, each without success.
https://learn.microsoft.com/en-us/dotnet/standard/security/walkthrough-creating-a-cryptographic-application
https://learn.microsoft.com/en-us/dotnet/api/system.security.cryptography.rijndaelmanaged?redirectedfrom=MSDN&view=netcore-3.1
Chained GZipStream/DeflateStream and CryptoStream (AES) breaks when reading
DeflateStream / GZipStream to CryptoStream and vice versa
https://learn.microsoft.com/en-us/dotnet/api/system.io.compression.gzipstream?redirectedfrom=MSDN&view=netcore-3.1#code-snippet-2
How to fix 'Found invalid data while decoding.'
Compression/Decompression string with C#

After ~2 days of investigating, i located my error.
I was calling rjndl.CreateEncryptor instead of rjndl.CreateDecryptor during the decryption portion... (Please tell me this type of $#!t happens to others too)
Once i finish testing i'll update my question code to serve as a nice example for anyone who lands here via google in the future.

AES-Encrypt-then-MAC a large file with .NET

I want to encrypt a large file (lets say 64 GB) in the most efficient way in .NET.
How I would implement this:
Create an instance of AesManaged to encrypt the stream of the file (read 64 GB)
Save this stream to disk (because it is to big to hold in memory) (write 64 GB)
Create an instance of HMACSHA512 to compute hash of the saved file (read 64 GB)
Save encrypted data with iv to disk (read & write 64 GB)
Simplified C# Code:
using (var aesManaged = new AesManaged())
{
using (var msEncrypt = File.OpenWrite(#"C:\Temp\bigfile.bin.tmp"))
{
using (var csEncrypt = new CryptoStream(msEncrypt, encryptor, CryptoStreamMode.Write))
{
File.OpenRead(#"C:\Temp\bigfile.bin").CopyTo(csEncrypt);
new MemoryStream(iv).CopyTo(csEncrypt);
}
}
}
using (var hmac = new HMACSHA512(hmacKey))
{
hmacHash = hmac.ComputeHash(File.OpenRead(#"C:\Temp\bigfile.bin.tmp"));
}
byte[] headerBytes;
using (var memoryStream = new MemoryStream())
{
var header = new Header
{
IV = iv,
HmacHash = hmacHash
};
Serializer.Serialize(memoryStream, header);
headerBytes = memoryStream.ToArray();
}
using (var newfile = File.OpenWrite(#"C:\Temp\bigfile.bin.enc"))
{
new MemoryStream(MagicBytes).CopyTo(newfile);
new MemoryStream(BitConverter.GetBytes(headerBytes.Length)).CopyTo(newfile);
new MemoryStream(headerBytes).CopyTo(newfile);
File.OpenRead(#"C:\Temp\bigfile.bin.tmp").CopyTo(newfile);
}
This implementation has the disadvantage that I created a second file and that I read multiple times 64 GB from disk.
Is the necessary? How to minimize disk IO and ram allocation?

I always get CryptoStreams wrong, so please excuse my pseudocode. The basic idea is to "chain" streams, so that plaintext gets copied to a cryptostream which does the encryption, which in turn writes data to a cryptostream that does the MACing, which then writes to plain old file stream:
using(var encryptedFileStream = File.OpenWrite("..."))
using(var macCryptoStream = new CryptoStream(encryptedFileStream, mac, CryptoStreamMode.Write))
using(var encryptCryptoStream = new CryptoStream(macCryptoStream, encryptor, CryptoStreamMode.Write))
using(var inputFileStream = File.OpenRead("..."))
inputFileStream.CopyTo(encryptCryptoStream);
This way, you only need a single pass through your 64 Gb.
Now, you'll have to somehow store the IV and MAC in the beginning of your encrypted file, so first "resize" it:
using(var encryptedFileStream = File.OpenWrite("..."))
{
var offset = YourMagicHeaderLength + IvLength + MacLength;
encryptedFileStream.SetLength(offset);
encryptedFileStream.Position = offset;
// The rest of the code goes here
}
and then, after encrypting and computing MAC, rewind to the very beginning and write them out.

Xml encrypted data error

I made the following code to insert data to an encrypted xml file, I used a memory stream for the decrypted file and load it in a XmlDocument to add the required data, then encrypted it again back to the same file.
public static void IN_EncryptFile(MemoryStream FLin,string Path)
{
UnicodeEncoding UE = new UnicodeEncoding();
byte[] key = UE.GetBytes(sKey);
string cryptFile = Path;
FileStream fsCrypt = new FileStream(cryptFile, FileMode.Open);
RijndaelManaged RMCrypto = new RijndaelManaged();
CryptoStream cs = new CryptoStream(fsCrypt,
RMCrypto.CreateEncryptor(key, key),
CryptoStreamMode.Write);
FLin.Position = 0;
int data;
while ((data = FLin.ReadByte()) != -1)
{
cs.WriteByte((byte)data);
}
FLin.Close();
FLin.Flush();
cs.Close();
fsCrypt.Close();
}
public static MemoryStream OUT_DecryptFile(string Path)
{
UnicodeEncoding UE = new UnicodeEncoding();
byte[] key = UE.GetBytes(sKey);
FileStream fsCrypt = new FileStream(Path, FileMode.Open);
RijndaelManaged RMCrypto = new RijndaelManaged();
CryptoStream cs = new CryptoStream(fsCrypt,
RMCrypto.CreateDecryptor(key, key),
CryptoStreamMode.Read);
MemoryStream fsOut = new MemoryStream();
int data;
while ((data = cs.ReadByte()) != -1)
fsOut.WriteByte((byte)data);
cs.Close();
fsCrypt.Close();
fsOut.Position = 0;
return fsOut;
}
public static void Add_Data_XML()
{
string XML_Pt = #"C:\Test.Trr";
XmlDocument XDt = new XmlDocument();
XDt.Load(Cryption.OUT_DecryptFile(XML_Pt));
/// Adding XMl data
MemoryStream Fnl = new MemoryStream();
XDt.Save(Fnl);
Cryption.IN_EncryptFile(Fnl, XML_Pt);
Fnl.Flush();
Fnl.Close();
}
I need an opinion about the code as it works fine for me but sometimes it generates some errors in the encrypted file which I still don't fully understand why, but when I dencrypt the xml file I find the added data are in the wrong format and placed after the main XMl node as follow :
<Main_Node>
</Main_Node>”ÇÛÏ”ö8—´Ú·…ï/1Ž"‹ÓÃåõ¶—QÝUŸy…¤Êç‹íîzR߆ô
nÃFçiŽÌm–FÆzÍW9 úv¤ï_øVO,ÈvÄ

Your issue is that you use MemoryStream to store the temporary data and you read from it while you can. MemoryStream has an internal buffer that is larger than the data you write to it usually. So if you keep reading from it, you may get garbage data out from it. I would recommend not reading byte by byte from it, or at least get the actual length of the data and read that many bytes.
You also first close the MemoryStream and then flush it. This is pointless.
Another problem could be because you use FileMode.Open and write less than the previously existed in the file. This will leave existing data after what you write and that will show as garbage. Using FileMode.Create would be the right way here, since it means "if there is no file with this name, create a new one. If there is a file, truncate it."
FileMode documentation
But if you surely always write more than you read, then this shouldn't be an issue.
Also I wouldn't say "it works fine for me" if it actually has issues.

I think you have an encoding issue. Try this
MemoryStream ms = new MemoryStream();
XDt.Save(ms);
StreamReader Fnl = new StreamReader(ms, Encoding.UTF8);

Add cleartext bytes to the beginning of a CryptoStream?

I have an interface defined like so:
public interface IEncryptionService
{
Stream Encrypt(Stream cleartext);
Stream Decrypt(Stream encrypted);
}
I am implementing this interface with an AesCryptoServiceProvider, but there's clearly a problem here. The IV (Initialization Vector) is not returned on the interface... so encrypting something would work fine, as long as I have no desire to decrypt it ever again. The Decrypt() method has no chance at all of working.
What I want to do is include the IV in cleartext at the beginning of the stream, then add the CryptoStream to it, so it is essentially encrypted data with a "header" that I could strip off and use for decrypting the stream.
So... how would I do that? I can create the CryptoStream easy enough, but it looks like this would encrypt the IV, which kinda defeats the purpose. I could load the CryptoStream into memory, prepend the IV, and then stream it out as a MemoryStream, but this would be really inefficient, and would die on large streams.
What is a good, secure, scalable practice for this?

Here is what I had in mind. See how you write the IV to the MemoryStream and then follow it with the crypto? Then when you want to decrypt, pull the IV off first in the same way.
Sorry, been a long time. This one is working. It should scale well if you don't cast ms toArray(); at the end. For example write to FileStream as you go and you should not need much memory at all. This is just to demo prepending the IV.
private byte[] encrypt(byte[] originalPlaintextBytes)
{
using (SymmetricAlgorithm algorithm = AesCryptoServiceProvider.Create())
{
algorithm.GenerateKey();
algorithm.GenerateIV();
byte[] iv = algorithm.IV;
byte[] key = algorithm.Key;
using (ICryptoTransform encryptor = algorithm.CreateEncryptor(key, iv))
{
using (MemoryStream ms = new MemoryStream())
using (CryptoStream cs = new CryptoStream(ms, encryptor,CryptoStreamMode.Write))
{
BinaryWriter bw = new BinaryWriter(ms);
bw.Write(iv);
cs.Write(originalPlaintextBytes, 0, originalPlaintextBytes.Length);
return ms.ToArray();
}
}
}
}
OK rather than edit the above code, here is a whole program that randomly creates a plaintext file of 1 megabyte. Then it encrypts it into ciphertext. Note that this program does not ever need 1 megabyte of memory in which to operate. It is completely scalable. Again, as before, this program is to demonstrate the concept, and you would do better with a readBuffer larger than 1 byte. But I did not want to create that and obscure the core answer. I hope this helps. I think it is exactly the kind of approach you need.
using System;
using System.IO;
using System.Security.Cryptography;
using System.Windows.Forms;
namespace SO_AES
{
public partial class Form1 : Form
{
Random ran = new Random();
public Form1()
{
InitializeComponent();
using (var file = File.Open("Plaintext.txt", FileMode.OpenOrCreate))
{
byte[] junkBytes = new byte[1000];
for (int i = 0; i < 1000; i++)
{
for (int j = 0; j < 1000; j++)
{
junkBytes[j] = (byte)ran.Next(0, 255);
}
file.Write(junkBytes, 0, junkBytes.Length);
}
}
using (var plainTextFile = File.Open("Plaintext.txt", FileMode.Open))
{
using (var cryptoTextFile = File.Open("Crypto.txt", FileMode.OpenOrCreate))
{
encrypt(plainTextFile, cryptoTextFile);
}
}
}
void encrypt(Stream inStream, Stream outStream)
{
using (SymmetricAlgorithm algorithm = AesCryptoServiceProvider.Create())
{
algorithm.GenerateKey();
algorithm.GenerateIV();
byte[] iv = algorithm.IV;
byte[] key = algorithm.Key;
using (ICryptoTransform encryptor = algorithm.CreateEncryptor(key, iv))
{
using (CryptoStream cs = new CryptoStream(outStream, encryptor, CryptoStreamMode.Write))
{
BinaryWriter bw = new BinaryWriter(outStream);
bw.Write(iv);
byte[] readBuffer = new byte[1];
BinaryReader br = new BinaryReader(inStream);
while (br.Read(readBuffer, 0, readBuffer.Length) != 0)
{
cs.Write(readBuffer, 0, 1);
}
}
}
}
inStream.Close();
outStream.Close();
}
}
}

Error on encryption and decryption.Couldn't find the cause of exceptions

I have following two methods.
1st method
//SymmetricEncryting
private byte[] SymmetricEncrypt()
{
try
{
//Get Byte Value
byte[] x= Encoding.Default.GetBytes("Test");
byte [] y;
//Create Symmetric Key Encription
RijndaelManaged rijndaelManaged = new RijndaelManaged();
//GetSymmetricPublicKey
_symmetricPublicKey = rijndaelManaged.Key;
//Get Symmetric Public IV
_symmetricPublicIv = rijndaelManaged.IV;
using (MemoryStream memoryStream = new MemoryStream(x))
{
//Start EncriptionProcess
var cryptoStream = new CryptoStream(memoryStream,
rijndaelManaged.CreateEncryptor
(_symmetricPublicKey,
_symmetricPublicIv),
CryptoStreamMode.Write);
cryptoStream.Write(x, 0, x.Length);
// Complete the encryption process
//cryptoStream.FlushFinalBlock();
y= memoryStream.ToArray();
}
return y;
}
catch (Exception)
{
throw;
}
}
2nd method
private string Decrypt(
byte[] y,
byte[] symmetricPublicKey,
byte[] symmtricPublicIv)
{
try
{
//Create the Key Container
CspParameters cspParameters = new CspParameters();
//Get the AsyPrivate and Public key from the Container
cspParameters.KeyContainerName = "Keys";
var rsaCryptoServiceProvider = new RSACryptoServiceProvider(cspParameters);
//Decrypt and get the Symmetric Public key
var decryptedSymmetricPubk = rsaCryptoServiceProvider.Decrypt(symmetricPublicKey, false);
//Decrypt and get the Symmetric Public IV
var decryptedSymmetricPubIv = rsaCryptoServiceProvider.Decrypt(symmtricPublicIv, false);
//Create RijndaelManaged object to do the Symmtric dycrption
RijndaelManaged rijndaelManaged = new RijndaelManaged();
//Create cryptostream using decrypted symmetric Public Key and IV
ICryptoTransform iCryptoTransform = rijndaelManaged.CreateDecryptor(decryptedSymmetricPubk,
decryptedSymmetricPubIv);
//Create a memory stream
using (MemoryStream memoryStream = new MemoryStream(y))
{
var cryptoStream = new CryptoStream(memoryStream, iCryptoTransform, CryptoStreamMode.Read);
byte[] z= new byte[y.Length];
cryptoStream.Read(z, 0, z.Length);
//cryptoStream.FlushFinalBlock();
//Convert byte array to string
var x= System.Text.Encoding.Default.GetString(z);
return x;
}
}
catch (Exception)
{
throw;
}
As you see in the code i am trying to encrypt a string using symmetric encryption.I encrypt the symmetric public key and Iv by using the asymmetric public key which i have already created.Then i am trying to decrypt the encrypted string .
Problem 1
What is the purpose of having cryptoStream.FlushFinalBlock(); on both encryption and decryption.As i learned from msdn it will end the processes running on the cyptostream
Problem 2
If i uncomment the line cryptoStream.FlushFinalBlock(); it throws an exception
"Memory stream is not expandable.". But if i comment the line it will work fine and return a byte array.
Problem 3
However the second method throws an exception "system.security.cryptography.cryptographicexception length of the data to decrypt is invalid ,on the execution of line cryptoStream.Read(z, 0, z.Length);
I couldn't find the actual cause of these errors on my debugging .Also i did some search on Google.But unfortunately i couldn't find any solution.Can any one please explain the answer?

You are encrypting using PKCS-padding (this is the default). AES/Rijndael is a block-cipher, which means that it only can encrypt blocks of 16 bytes at a time. To allow block-cipher to encrypt data of arbitrary sizes we use a padding algorithm. PKCS-padding works by adding 1-16 bytes at the end when encrypting and removing them when decrypting. The length of the padding is encoded in the padding itself.
You need the FlushFinalBlock when encrypting to let the CryptoStream know that there is no more incoming data and it should add the padding. It is not necessary and should not be used when you are using the CryptoStream in Read-mode.
The first exception come because you are using the plaintext-array as a backing store for the MemoryStream. Because of the padding the encryption will be larger than the plaintext.
The second exception is because you removed the FlushFinalBlock statement and because the MemoryStream is not allowed to resize to make an array of the correct length. The encrypted data should always be a multiple of 16 bytes, but since the MemoryStream will reuse x, y will have the same length as x, which is not always a valid length.
The solution is:
Use FlushFinalBlock in SymmetricEncrypt.
Replace using (MemoryStream memoryStream = new MemoryStream(x)) and using (MemoryStream memoryStream = new MemoryStream(y)) with using (MemoryStream memoryStream = new MemoryStream()). This will allow the MemoryStreams to resize freely.

Oddly, it works for me to just do a write operation when de-crypting as well. Something like:
var decryptMemoryStream = new MemoryStream();
var decryptStream = new CryptoStream(decryptMemoryStream, iCryptoTransform , CryptoStreamMode.Write);
//write the unencrypted data array to the stream
decryptStream.Write(y, 0, y.Length);
decryptStream.Flush();
decryptStream.Close();
var decryptedData = decryptMemoryStream.ToArray();