We Keep Coding

File.ReadAllBytes takes a lot of RAM, is there a better way?

I am busy working with an encryption lib, and I want to encrypt large files (with AES GCM). Currently I have this for encrypting files, after writing a temp file, Chipher, from the CryptoStream:
byte[] Chiper = File.ReadAllBytes(BUFFER_PATH);
// Retrieve tag and create array to hold encrypted data.
byte[] AuthTag = encryptor.GetTag();
byte[] encrypted = new byte[Chiper.Length + aes.IV.Length + AuthTag.Length];
// Set needed data in byte array.
aes.IV.CopyTo(encrypted, 0);
AuthTag.CopyTo(encrypted, IV_LENGTH);
Chiper.CopyTo(encrypted, IV_LENGTH + TAG_LENGTH);
File.WriteAllBytes(END_PATH, encrypted);
This function works fine, however it takes a lot of RAM depending on the filesize. Is there a better way though this? I tried using a FileStream though it starts conflicting with my code. Is there a way to use less, or no memory to save Chiper(byte[])?

It appears that you're trying to write a composite file that has three pieces of information - the tag, IV and cipher text. Given that you can't get the tag value until after the encryption completes, you are trying to composite the data after encryption completes.
The problem comes in when you attempt to load a large encrypted file into memory. Fortunately, streams provide a nice simple solution for this:
byte[] authTag = encryptor.GetTag();
using (var tempfile = File.OpenRead(BUFER_PATH))
using (var outstream = File.Create(END_PATH))
{
// write tag
outstream.Write(authTag, 0, authTag.Length);
// write IV
outstream.Write(aes.IV, 0, aes.IV.Length);
// copy data from source file to output file
tempfile.CopyTo(outstream);
}
On the other hand you could also write the data straight to the output file if you know ahead of time what size the tag and IV are going to be. Just allocate space for the tag value at the start of the file and come back and write it in after the fact. That saves you having to use a temporary file.

AzCopy Ignoring or recreating Checksum

I'm using the Azure tool AzCopy to export data from table storage, modify the exported data, and then import the data into another table storage table. I'm using the following command to export:
AzCopy /Source:https://MYSERVER/MYTABLE/ /SourceKey:SOURCEKEY /Dest:C:\migration /Manifest:MYTABLE
Since you cannot add a filter for the export, I'm filtering the data post-export, remove data from the JSON as necessary. I'm then using the following command to import this data to another server:
AzCopy/Source:C:\export /Dest:https://MYOTHERSERVER/MYTABLE /DestType:Table /DestKey:DESTKEY /Manifest:MYTABLE EntityOperation:InsertOrReplace
These operations work fine when I do not manipulate the JSON file. When I do, however, the contents of the file are, of course, changed and the checksum in the manifest file no longer matches. When I go to do the import, I get a "file is corrupt" message.
Here is what the manifest file looks like:
"Version":2,"PayloadFormat":"Json","Checksum":5500917691400439101,"AccountName":"SERVER","TableName":"MYTABLE","Timestamp":"2017-08-25T14:10:53.7489755Z","SplitSize":0,"TotalDataFiles":1}
How can I get AzCopy to either not validate the checksum or replace the checksum?
I've tried the following code to recreate the checksum, but when I do on the original JSON, it does not match:
var md5Hash = getFileHash(file);
var checksum = convertHash(md5Hash);
private byte[] getFileHash(string filePath)
{
using (var md5 = MD5.Create())
{
using (var stream = File.OpenRead(filePath))
{
return md5.ComputeHash(stream);
}
}
}
private string convertHash(byte[] data)
{
var algorithm = MD5.Create();
var result = BitConverter.ToUInt64(data,0);
return result.ToString();
}
This returns 4500814390503865422.

AzCopy doesn't support skipping checksum validation during table import for now. BTW, the checksum recorded in manifest file is actually CRC rather than MD5, and it's calculated by aggregating CRC of all exported files rather than the single manifest file.

Check corrupt file before copying to another file

I want to check a file is corrupt or not before copying to another file which is actually a backup of 1st one and will be restored if something goes wrong with original file.
System.IO.File.Copy(FileA, FileB, true);
Sometimes my original file get corrupted and as i have no check for corruption while copying i also corrupt my backup file.
Any help will be appreciated.
Thanks,

You can check with MD5
using (var md5 = MD5.Create())
{
using (var stream = File.OpenRead(filename))
{
return md5.ComputeHash(stream);
}
}
font: Calculate MD5 checksum for a file

How to check if a file has been modified in c#

I want to prompt a user to save a file when some modification has been done to it but the problem is i can't for the life of me do that.
Some people have suggested using FileInfo class but it only gives you the lastWriteTime, LastAccessTime and CreationTime.
I would like to use FileInfo class rather than FileSystemWatcher to check for modifications but how?
Example: Say a user has edited a file, within my program, that they loaded and clicks EXIT, i want a way to check whether any modifications were done on the file. If none, exit. If some, prompt user to save the file. So how do i check for modifications on that FILE?

The easiest way is to calculate the MD5 hash of the file and compare to the original MD5 hash and if these two don't match the file was modified...
using (var md5 = new MD5CryptoServiceProvider())
{
var buffer = md5.ComputeHash(File.ReadAllBytes(filename));
var sb = new StringBuilder();
for (var i = 0; i < buffer.Length; i++)
{
sb.Append(buffer[i].ToString("x2"));
}
return sb.ToString();
}

Here are some examples of how to use the File or FileInfo class to get the LastWriteTime.
http://www.csharp-examples.net/file-creation-modification-time/
I would store the timestamp of the file when you load it, then compare it to the File.GetLastWriteTime() to see if the file has been saved since then. If the file was modified by an outside source, you can give the user the option to discard their changes and reload the file, or save their changes to a new file.

Generate checksum without opening the file

Is there a way to calculate the checksum of a file that is readonly?
The only examples I have seen uses an algorithm like this
public string GetChecksum()
{
FileStream file = new FileStream(_filePath, FileMode.Open);
MD5 md5 = new MD5CryptoServiceProvider();
byte[] retVal = md5.ComputeHash(file);
file.Close();
StringBuilder sb = new StringBuilder();
foreach (byte t in retVal)
{
sb.Append(retVal[1].ToString("x2"));
}
return sb.ToString();
}

You can open a file even if it's readonly.
It is not possible to generate a checksum without opening the file, since you can't read a file without opening it.
You should pass FileAccess.Read to open it as read-only.
Also, you should generate checksums using SHA512, not MD5.

According to the documentation, the FileStream constructor you are using opens the file for read/write. Use an overload that specifies FileAccess.Read.
The constructor is given read/write
access to the file, and it is opened
sharing Read access

You cannot generate a checksum without reading the entire file.
Generally, readonly files can be opened. There might be file or folder permissions that prevent a given user from opening the file.

Well, no. You have to read a file to do anything with what's in it. But you're opening with Generic access when you probably want FileStream(_filePath,FileAccess.Read,true,4096,true); to open it read-only. StreamReader will do this automatically.