We Keep Coding

Unable to resolve compression, encryption, decryption, decompression stream implementation

I've been fighting with chained using statements, and am unable to resolve the latest in a long line of implementation issues. I need to compress, then encrypt and append the generated IV to the selected file. This all appears to work correctly, however i'm unable to unwind the process. After looking at several similar stack postings and articles i'm still unable to get it to work and am now after more direct assistance.
The latest thrown error is System.IO.InvalidDataException: 'Found invalid data while decoding.' It appears that the decryption stream isn't functioning as intended and that's throwing the decompression stream out of wack.
byte[] key;
byte[] salt;
const int keySize = 256;
const int blockSize = keySize;
byte[] iv = new byte[blockSize / 8];//size to bits
RijndaelManaged rjndl;
RNGCryptoServiceProvider cRng;
void InitializeCryptor() {
//Temporarily define the salt & key
salt = Encoding.UTF8.GetBytes("SaltShouldBeAtLeast8Bytes");
key = new Rfc2898DeriveBytes("MyL0ngPa$$phra$e", salt, 4).GetBytes(keySize / 8);
//Initialize the crypto RNG generator
cRng = new RNGCryptoServiceProvider();
// Create instance of Rijndael (AES) for symetric encryption of the data.
rjndl = new RijndaelManaged();
rjndl.KeySize = keySize;
rjndl.BlockSize = blockSize;
rjndl.Mode = CipherMode.CBC;
}
void CompressAndEncryptFile(string relativeFilePath, string fileName) {
//Create a unique IV each time
cRng.GetBytes(iv);
//Create encryptor
rjndl.Key = key;
rjndl.IV = iv;
ICryptoTransform encryptor = rjndl.CreateEncryptor(rjndl.Key, rjndl.IV);
//Create file specific output sub-directory
Directory.CreateDirectory(Path.Combine(outputPath, relativeFilePath));
//Read and compress file into memory stream
using (FileStream readStream = File.OpenRead(Path.Combine(initialpath, relativeFilePath, fileName)))
using (FileStream writeStream = new FileStream(Path.Combine(outputPath, relativeFilePath, fileName + ".dat"), FileMode.Create))
using (CryptoStream encryptStream = new CryptoStream(writeStream, encryptor, CryptoStreamMode.Write))
using (DeflateStream compStream = new DeflateStream(encryptStream, CompressionLevel.Optimal)) {
//Write the following to the FileStream for the encrypted file:
// - length of the IV
// - the IV
byte[] ivSize = BitConverter.GetBytes(rjndl.IV.Length);
writeStream.Write(ivSize, 0, 4);
writeStream.Write(rjndl.IV, 0, rjndl.BlockSize / 8);
readStream.CopyTo(compStream);
}
}
void DecryptAndDecompressFile(string relativeFilePath) {
string outputPath = Path.Combine(initialpath, "Unpack");
Directory.CreateDirectory(outputPath);
using (FileStream readStream = new FileStream(Path.Combine(initialpath, manifestData.version, relativeFilePath + ".dat"), FileMode.Open)) {
byte[] tmpLength = new byte[4];
//Read length of IV
readStream.Seek(0, SeekOrigin.Begin);
readStream.Read(tmpLength, 0, 3);
int ivLength = BitConverter.ToInt32(tmpLength, 0);
byte[] readIv = new byte[ivLength];
//Read IV
readStream.Seek(4, SeekOrigin.Begin);
readStream.Read(readIv, 0, ivLength);
rjndl.IV = readIv;
//Start at beginning of encrypted data
readStream.Seek(4 + ivLength, SeekOrigin.Begin);
//Create decryptor
ICryptoTransform decryptor = rjndl.CreateEncryptor(key, readIv);
using (CryptoStream decryptStream = new CryptoStream(readStream, decryptor, CryptoStreamMode.Read))
using (DeflateStream decompStream = new DeflateStream(decryptStream, CompressionMode.Decompress))
using (FileStream writeStream = new FileStream(Path.Combine(outputPath, relativeFilePath), FileMode.Create)) {
decompStream.CopyTo(writeStream);
}
}
}
For those who like to point to other similar stack questions and vote to close/duplicate without offering support, the following are the threads, and posts I've worked through first, each without success.
https://learn.microsoft.com/en-us/dotnet/standard/security/walkthrough-creating-a-cryptographic-application
https://learn.microsoft.com/en-us/dotnet/api/system.security.cryptography.rijndaelmanaged?redirectedfrom=MSDN&view=netcore-3.1
Chained GZipStream/DeflateStream and CryptoStream (AES) breaks when reading
DeflateStream / GZipStream to CryptoStream and vice versa
https://learn.microsoft.com/en-us/dotnet/api/system.io.compression.gzipstream?redirectedfrom=MSDN&view=netcore-3.1#code-snippet-2
How to fix 'Found invalid data while decoding.'
Compression/Decompression string with C#

After ~2 days of investigating, i located my error.
I was calling rjndl.CreateEncryptor instead of rjndl.CreateDecryptor during the decryption portion... (Please tell me this type of $#!t happens to others too)
Once i finish testing i'll update my question code to serve as a nice example for anyone who lands here via google in the future.

Encrypt and compress FileStream and save to same file

I have a method to compress and encrypt a stream.
private static void CompressThenEncrypt(string inputFileName, ICryptoTransform encryptor)
{
using (var inputFileStream = new FileStream(inputFileName, FileMode.Open, FileAccess.ReadWrite))
using (var cryptoStream = new CryptoStream(inputFileStream, encryptor, CryptoStreamMode.Write))
using (var zipStream = new GZipStream(cryptoStream, CompressionMode.Compress))
{
inputFileStream.CopyTo(zipStream);
}
}
Which does work, but it appends compressed and encrypted data to raw one I have in that file.
So if my file contains:
kkk
Then after compression and encryption it does look for example like that:
kkkㆆ鬁⠕⟶ꏙᇚ셑襜㷡ꕢ束৘㺝娥☪
Do anyone have any idea how to overwrite new data?

I suggest:
Write to a new file
Delete the old file
Rename the new file to the old file.

Write to a temporary MemoryStream. Something like:
private static void CompressThenEncrypt(string inputFileName, ICryptoTransform encryptor)
{
using (var fileStream = new FileStream(inputFileName,FileMode.Open, FileAccess.ReadWrite))
using (var memoryStream = new MemoryStream())
using (var cryptoStream = new CryptoStream(memoryStream, encryptor, CryptoStreamMode.Write))
using (var zipStream = new GZipStream(cryptoStream, CompressionMode.Compress))
{
fileStream.CopyTo(zipStream);
cryptoStream.FlushFinalBlock();
fileStream.SetLength(0);
memoryStream.Position = 0;
memoryStream.CopyTo(fileStream);
}
}
I think that should work, from the top of my head. Not sure you would have to set the length, and I'm not sure you can get away with not resetting the position of the MemoryStream. I haven't tested this code. If you are concerned about memory usage, you can always buffer the usage, but that would require more complicated code.

Decrypting CryptoStream into MemoryStream

I have written a process where a file is encrypted and uploaded to Azure, then the download process has to be decrypted which is what fails with a "Padding is invalid and cannot be removed" error, or a "Length of the data to decrypt is invalid." error.
I've tried numerous solutions online, including C# Decrypting mp3 file using RijndaelManaged and CryptoStream, but none of them seem to work and I end up just bouncing back and forth between these two errors. The encryption process uses the same key/IV pair that decryption uses, and since it will decrypt a portion of the stream I feel like that's working fine - it just ends up dying with the above errors.
Here is my code, any ideas? Please note that the three variants (cryptoStream.CopyTo(decryptedStream), do {} and while) aren't run together - they are here to show the options I've already tried, all of which fail.
byte[] encryptedBytes = null;
using (var encryptedStream = new MemoryStream())
{
//download from Azure
cloudBlockBlob.DownloadToStream(encryptedStream);
//reset positioning for reading it back out
encryptedStream.Position = 0;
encryptedBytes = encryptedStream.ConvertToByteArray();
}
//used for the blob stream from Azure
using (var encryptedStream = new MemoryStream(encryptedBytes))
{
//stream where decrypted contents will be stored
using (var decryptedStream = new MemoryStream())
{
using (var aes = new RijndaelManaged { KeySize = 256, Key = blobKey.Key, IV = blobKey.IV })
{
using (var decryptor = aes.CreateDecryptor())
{
//decrypt stream and write it to parent stream
using (var cryptoStream = new CryptoStream(encryptedStream, decryptor, CryptoStreamMode.Read))
{
//fails here with "Length of the data to decrypt is invalid." error
cryptoStream.CopyTo(decryptedStream);
int data;
//fails here with "Length of the data to decrypt is invalid." error after it loops a number of times,
//implying it is in fact decrypting part of it, just not everything
do
{
data = cryptoStream.ReadByte();
decryptedStream.WriteByte((byte)cryptoStream.ReadByte());
} while (!cryptoStream.HasFlushedFinalBlock);
//fails here with "Length of the data to decrypt is invalid." error after it loops a number of times,
//implying it is in fact decrypting part of it, just not everything
while ((data = cryptoStream.ReadByte()) != -1)
{
decryptedStream.WriteByte((byte)data);
}
}
}
}
//reset position in prep for reading
decryptedStream.Position = 0;
return decryptedStream.ConvertToByteArray();
}
}
One of the comments mentioned wanting to know what ConvertToByteArray is, and it's just a simple extension method:
/// <summary>
/// Converts a Stream into a byte array.
/// </summary>
/// <param name="stream">The stream to convert.</param>
/// <returns>A byte[] array representing the current stream.</returns>
public static byte[] ConvertToByteArray(this Stream stream)
{
byte[] buffer = new byte[16 * 1024];
using (MemoryStream ms = new MemoryStream())
{
int read;
while ((read = stream.Read(buffer, 0, buffer.Length)) > 0)
{
ms.Write(buffer, 0, read);
}
return ms.ToArray();
}
}
The code never reaches this though - it dies before I can ever get it to this point.

After a lot of back and forth from various blogs, I found I actually had a couple of errors in the above code that were nailing me. First, the encryption process was incorrectly writing the array - it was wrapped with a CryptoStream instance, but wasn't actually utilizing that so I was writing the unencrypted data to Azure. Here is the proper route to go with this (fileKey is part of a custom class I created to generate Key/IV pairs, so wherever that is referenced can be changed to the built-in process from RijndaelManaged or anything else you'd utilize for coming up with a key/IV pair):
using (var aes = new RijndaelManaged { KeySize = 256, Key = fileKey.Key, IV = fileKey.IV })
{
using (var encryptedStream = new MemoryStream())
{
using (ICryptoTransform encryptor = aes.CreateEncryptor())
{
using (CryptoStream cryptoStream = new CryptoStream(encryptedStream, encryptor, CryptoStreamMode.Write))
{
using (var originalByteStream = new MemoryStream(file.File.Data))
{
int data;
while ((data = originalByteStream.ReadByte()) != -1)
cryptoStream.WriteByte((byte)data);
}
}
}
var encryptedBytes = encryptedStream.ToArray();
return encryptedBytes;
}
}
Second, since my encryption process involves multiple steps (three total keys per file - container, filename and file itself), when I tried to decrypt, I was using the wrong key (which is seen above when I referenced blobKey to decrypt, which was actually the key used for encrypting the filename and not the file itself. The proper decryption method was:
//used for the blob stream from Azure
using (var encryptedStream = new MemoryStream(encryptedBytes))
{
//stream where decrypted contents will be stored
using (var decryptedStream = new MemoryStream())
{
using (var aes = new RijndaelManaged { KeySize = 256, Key = blobKey.Key, IV = blobKey.IV })
{
using (var decryptor = aes.CreateDecryptor())
{
//decrypt stream and write it to parent stream
using (var cryptoStream = new CryptoStream(encryptedStream, decryptor, CryptoStreamMode.Read))
{
int data;
while ((data = cryptoStream.ReadByte()) != -1)
decryptedStream.WriteByte((byte)data);
}
}
}
//reset position in prep for reading
decryptedStream.Position = 0;
return decryptedStream.ConvertToByteArray();
}
}
I had looked into the Azure Encryption Extensions (http://www.stefangordon.com/introducing-azure-encryption-extensions/), but it was a little more local file-centric than I was interested - everything on my end is streams/in-memory only, and retrofitting that utility was going to be more work than it was worth.
Hopefully this helps anyone looking to encrypt Azure blobs with zero reliance on the underlying file system!

Bit late to the party, but in case this is useful to someone who finds this thread:
The following works well for me.
internal static byte[] AesEncryptor(byte[] key, byte[] iv, byte[] payload)
{
using (var aesAlg = Aes.Create())
{
aesAlg.Mode = CipherMode.CBC;
aesAlg.Padding = PaddingMode.PKCS7;
var encryptor = aesAlg.CreateEncryptor(key, iv);
var encrypted = encryptor.TransformFinalBlock(payload, 0, payload.Length);
return iv.Concat(encrypted).ToArray();
}
}
and to decrypt:
internal static byte[] AesDecryptor(byte[] key, byte[] iv, byte[] payload)
{
using (var aesAlg = Aes.Create())
{
aesAlg.Mode = CipherMode.CBC;
aesAlg.Padding = PaddingMode.PKCS7;
var decryptor = aesAlg.CreateDecryptor(aesAlg.Key, aesAlg.IV);
return decryptor.TransformFinalBlock(payload, 0, payload.Length);
}
}
this works for encrypting/decrypting both fixed length hex strings when decoded from hex to byte[] as well as utf8 variable length strings when decoded using Encoding.UTF8.GetBytes().

C# - AES Decryption performance slow

I've currently got a folder full of 1280x720 AES encrypted bitmaps.
I'm trying to create a player to loop through the folder decrypt the images and show them in a image box (at reasonable speed)
Importantly I don't want the files to be decrypted to the drive then played. I want to decrypt them in memory only.
Currently the decryption takes about 100ms per image (frame). But I would like to try and get this down to about 10ms if possible.
The above was profiled on a 3.0ghz iCore7
Currently I'm running everything on the UI thread . I thought maybe if I multi-threaded the decrypting I could probably get the speed I wanted, then I will have to store a lot in memory. But I would rather see if I could just make the actual decryption faster.
Here is the decryption function:
private byte[] DecryptFile(string inputFile, string skey)
{
MemoryStream output1 = new MemoryStream();
// ok for tests..
Rfc2898DeriveBytes k2 = new Rfc2898DeriveBytes(skey, new byte[] { 10, 10, 10, 10, 10, 10, 10, 10 });
try
{
using (RijndaelManaged aes = new RijndaelManaged())
{
byte[] key = k2.GetBytes(16);
/* This is for demostrating purposes only.
* Ideally yu will want the IV key to be different from your key and you should always generate a new one for each encryption in other to achieve maximum security*/
byte[] IV = k2.GetBytes(16);
byte[] cript = File.ReadAllBytes(inputFile);
using (MemoryStream fsCrypt = new MemoryStream(cript))
{
using (ICryptoTransform decryptor = aes.CreateDecryptor(key, IV))
{
using (CryptoStream cs = new CryptoStream(fsCrypt, decryptor, CryptoStreamMode.Read))
{
cs.CopyTo(output1);
}
}
}
}
}
catch (Exception ex)
{
MessageBox.Show(ex.Message);
}
return output1.ToArray() ;
}
Is there a more efficient way to decrypt than the above function?

You can use the AesCryptoServiceProvider which is faster.
In contrast to RijndaelManaged which is a pure managed implementation, AesCryptoServiceProvider uses the Windows Crypto API.

Rfc2898DeriveBytes is designed to be intentionally slow, it used to slow down brute force hacking attempts. Since every file looks like it is using the same key and IV (btw, to address the IV issue you say in the comments, just store the IV has the first bytes in the file itself. The IV does not need to be kept secret, only the key does)
Here is a updated version with a few other tweaks too.
private IEnumerable<byte[]> DecryptFiles(IEnumerable<string> inputFiles, string skey)
{
//Only performing the key calculation once.
Rfc2898DeriveBytes k2 = new Rfc2898DeriveBytes(skey, new byte[] { 10, 10, 10, 10, 10, 10, 10, 10 });
byte[] key = k2.GetBytes(16)
foreach(var inputFile in inputFiles)
{
yield return DecryptFile(inputFile, key);
}
}
private byte[] DecryptFile(string inputFile, byte[] key)
{
var output1 = new MemoryStream();
try
{
//If you are going to use AES, then use AES, also the CSP is faster than the managed version.
using (var aes = new AesCryptoServiceProvider ())
{
//No need to copy the file in to memory first, just read it from the hard drive.
using(var fsCrypt = File.OpenRead(inputFile))
{
//Gets the IV from the header of the file, you will need to modify your Encrypt process to write it.
byte[] IV = GetIV(fsCrypt);
//You can chain consecutive using statements like this without brackets.
using (var decryptor = aes.CreateDecryptor(key, IV))
using (var cs = new CryptoStream(fsCrypt, decryptor, CryptoStreamMode.Read))
{
cs.CopyTo(output1);
}
}
}
}
catch (Exception ex)
{
MessageBox.Show(ex.Message);
}
return output1.ToArray();
}
//This function assumes you wrote a 32 bit length then the array that was the read length long.
private static byte[] GetIV(Stream fileStream)
{
var reader = new BinaryReader(fileStream);
var keyLength = reader.ReadInt32();
return reader.ReadBytes(keyLength);
}

If you want to "roll your own", you might want to take a look at this paper describing new instructions on the i7 that are geared towards optimal AES performance. If your C# library doesn't take advantage of those you might get a nice speed boost.
From this paper, it seems that the best you can hope to achieve with a 256 byte key is about 0.32 cycles / byte - for a 1MB file and a 3 GHz processor using all cores with hyperthreading, that would be roughly 0.1 ms per file. That's 1000x faster than you are seeing - so yes, you seem to be well off the "top speed" mark. Same paper claims about 6x slower for single core - still much faster than you are seeing.
I would go looking for another library before writing my own, though. Take a look at the one that Intel provides: http://software.intel.com/en-us/articles/download-the-intel-aesni-sample-library

This is what I use in my system...not sure if it will be faster or not. It just seems like you're doing an un-necessary amount of reading and copying:
Public Shared Function ReadImageAES(ByVal FileName As String) As Image
Dim img As Image = Nothing
Try
Using fs As New FileStream(FileName, FileMode.Open)
Using cs As New CryptoStream(fs, Crypto.AES.CreateDecryptor, CryptoStreamMode.Read)
img = Image.FromStream(cs)
End Using
End Using
Catch ex As Exception
img = Nothing
'Debug.Print("ReadImageAES()failed: " & ex.ToString)
End Try
Return img
End Function
Public Shared Sub WriteImageAES(ByVal FileName As String, ByVal img As Image, Optional ByVal NewUserKey As String = Nothing)
If Not IsNothing(img) Then
Try
If File.Exists(FileName) Then
File.Delete(FileName)
End If
Using fs As New FileStream(FileName, FileMode.OpenOrCreate)
Dim Key() As Byte
If IsNothing(NewUserKey) Then
Key = Crypto.AES.Key
Else
Key = Crypto.SHA256Hash(NewUserKey)
End If
Using cs As New CryptoStream(fs, Crypto.AES.CreateEncryptor(Key, Crypto.AES.IV), CryptoStreamMode.Write)
Dim bmp As New Bitmap(img)
bmp.Save(cs, System.Drawing.Imaging.ImageFormat.Jpeg)
bmp.Dispose()
cs.FlushFinalBlock()
End Using
End Using
Catch ex As Exception
'Debug.Print("WriteImageAES() Failed: " & ex.ToString)
End Try
Else
MessageBox.Show(FileName, "GetImage() Failed")
End If
End Sub
Here's my Crypto class, which was originally written to only target .Net 2.0:
Imports System.IO
Imports System.Text
Imports System.Security.Cryptography
Public Class Crypto
Private Shared _UserKey As String = ""
Private Shared _SHA256 As New SHA256Managed
Private Shared _AES As New RijndaelManaged
Private Const _IV As String = "P5acZMXujMRdmYvFXYfncS7XhrsPNfHkerTnWVT6JcfcfHFDwa" ' <--- this can be anything you want
Public Shared Property UserKey() As String
Get
Return Crypto._UserKey
End Get
Set(ByVal value As String)
Crypto._UserKey = value
Crypto._AES.KeySize = 256
Crypto._AES.BlockSize = 256
Crypto._AES.Key = Crypto.SHA256Hash(Crypto._UserKey)
Crypto._AES.IV = Crypto.SHA256Hash(Crypto._IV)
Crypto._AES.Mode = CipherMode.CBC
End Set
End Property
Public Shared Function SHA256Hash(ByVal value As String) As Byte()
Return Crypto._SHA256.ComputeHash(ASCIIEncoding.ASCII.GetBytes(value))
End Function
Public Shared ReadOnly Property AES() As RijndaelManaged
Get
Return Crypto._AES
End Get
End Property
End Class

I got it almost two time faster when using byte array strictly as a output instead of yours MemoryStream.
My code sample:
byte[] ds = new byte[data.Length];
byte[] decryptedData;
using (Aes aes = CreateAes(key, iv, cipherMode, paddingMode))
{
using (ICryptoTransform decryptor = aes.CreateDecryptor(aes.Key, aes.IV))
{
int i = 0;
using (MemoryStream msDecrypt = new MemoryStream(data))
{
using (CryptoStream csDecrypt = new CryptoStream(msDecrypt, decryptor, CryptoStreamMode.Read))
{
int k;
while ((k = csDecrypt.ReadByte()) != -1)
{
ds[i++] = (byte)k;
}
}
}
decryptedData = new byte[i];
Buffer.BlockCopy(ds, 0, decryptedData, 0, i);
}
}
return decryptedData;

Any alternatives for this code to know the byte count?

okay so I have this code for decrypting files
public static byte[] DecryptFile(string inputFile, string skey)
{
RijndaelManaged aes = new RijndaelManaged();
byte[] key = ASCIIEncoding.UTF8.GetBytes(skey);
using (FileStream fsCrypt = new FileStream(inputFile, FileMode.Open))
{
using (CryptoStream cs =
new CryptoStream(fsCrypt, aes.CreateDecryptor(key, key),
CryptoStreamMode.Read))
{
using (BinaryReader reader = new BinaryReader(cs))
{
byte[] str = reader.ReadBytes(Convert.ToInt32(cs.Length));
reader.Close();
cs.Close();
return (str);
}
}
}
}
}
NOW i've got a problem with it, i can't determine the byte length! I tried
cs.Length
but it says the Stream doesn't support seeking (something like tht)
I also tried counting the bytes of the file by
File.ReadAllBytes(encrypted_file_path).Length
but it says the file is in use...it is indeed in use because of the FileStream fsCrypt
for the meantime I replaced cs.Length with some large integer to make it work..like 1000000..the maximum integer that doesn't cause any exception..it does work that way.

You cannot know the length until after you decrypt the entire file.
Therefore, you need to start with a small array, and make it bigger as it gets full.
The MemoryStream class does just that; you can just cs.CopyTo() into a new MemoryStream and call ToArray().