How can i use Calculate MD5 Hash as Async?
I have this code:
static string CalculateMD5(string filename)
{
using (var md5 = MD5.Create())
{
using (var stream = File.OpenRead(filename))
{
var hash = md5.ComputeHash(stream);
return BitConverter.ToString(hash).Replace("-", "").ToUpperInvariant();
}
}
}
I want to hash a file of 1GB and then it freezes my app for 20 sec.
As you are reading from a file you can use IO-bound asynchronicity. Unfortunarely neither the File class nor the MD5 class support it out of the box. But it can be manually done:
static async Task<string> CalculateMD5Async(string filename)
{
using (var md5 = System.Security.Cryptography.MD5.Create())
{
using (var stream = new FileStream(filename, FileMode.Open, FileAccess.Read, FileShare.Read, 4096, true)) // true means use IO async operations
{
byte[] buffer = new byte[4096];
int bytesRead;
do
{
bytesRead = await stream.ReadAsync(buffer, 0, 4096);
if (bytesRead > 0)
{
md5.TransformBlock(buffer, 0, bytesRead, null, 0);
}
} while (bytesRead > 0);
md5.TransformFinalBlock(buffer, 0, 0);
return BitConverter.ToString(md5.hash).Replace("-", "").ToUpperInvariant();
}
}
For performance reasons you may want to increase the buffer size.
Try this:
private async void CalculateMd5_Click(object sender, EventArgs e)
{
var filePath = "Path/to/your/file";
var md5hash = await CalculateMd5ForFileAsync(filePath);
}
private Task<string> CalculateMd5ForFileAsync(string filePath)
{
return new Task<string>(() => ComputeMd5HashForFile(filePath));
}
private string ComputeMd5HashForFile(string filePath)
{
using (var md5 = MD5.Create())
{
using (var stream = File.OpenRead(filePath))
{
var hash = md5.ComputeHash(stream);
return BitConverter.ToString(hash).Replace("-", "").ToUpperInvariant();
}
}
}
Related
I have to prepare zip file and secure it with password.
I'm using C# and SharpZipLib library.
I have to return the created file using WebApi. The code below does not work - I can open file without providing password. What do I do wrong?
[HttpGet]
[Route("zip")]
public async Task ZipFile()
{
Response.StatusCode = 200;
Response.Headers.Add("ContentDisposition", $"attachment; filename=\"{Path.GetFileName("z.zip")}\"");
Response.Headers.Add("ContentType", "application/octet-stream");
compressDirectory("C:\\temp\\files");
}
private void compressDirectory(string DirectoryPath, int CompressionLevel = 9)
{
string[] filenames = Directory.GetFiles(DirectoryPath);
using (ZipOutputStream OutputStream = new ZipOutputStream(Response.Body))
{
OutputStream.SetLevel(CompressionLevel);
byte[] buffer = new byte[4096];
for (int i=0; i<filenames.Length; i++)
{
ZipEntry entry = new ZipEntry($"{i}\\" +Path.GetFileName(filenames[i]));
entry.DateTime = DateTime.Now;
OutputStream.PutNextEntry(entry);
using (FileStream fs = System.IO.File.OpenRead(filenames[i]))
{
int sourceBytes;
do
{
sourceBytes = fs.Read(buffer, 0, buffer.Length);
OutputStream.Write(buffer, 0, sourceBytes);
} while (sourceBytes > 0);
}
}
OutputStream.Password = "123";
OutputStream.Finish();
OutputStream.Close();
}
}
So, i'm trying to convert a large byte array into it's base64 encoded variant. But no matter what i try, it seems to freeze up my UI every time it runs.
This is what i've got currently:
private async void TxtOutput_DragDrop(object sender, DragEventArgs e)
{
string outputText = String.Empty;
if (e.Data.GetDataPresent(DataFormats.FileDrop))
{
string[] path = (string[])e.Data.GetData(DataFormats.FileDrop);
byte[] fileBytes = File.ReadAllBytes(path[0]);
txtOutput.Text = await Task.Run(() => {return Convert.ToBase64String(fileBytes);});
_ = fileBytes;
_ = path;
}
}
So, the line that freezes everything up is:
txtOutput.Text = await Task.Run(() => {return Convert.ToBase64String(fileBytes);});
File.ReadAllBytes(path[0])
Could be a bottle neck you can use async operation for read files
Here is an example how to read file async
public async Task ProcessReadAsync()
{
string filePath = #"temp2.txt";
if (File.Exists(filePath) == false)
{
Debug.WriteLine("file not found: " + filePath);
}
else
{
try
{
string text = await ReadTextAsync(filePath);
Debug.WriteLine(text);
}
catch (Exception ex)
{
Debug.WriteLine(ex.Message);
}
}
}
private async Task<string> ReadTextAsync(string filePath)
{
using (FileStream sourceStream = new FileStream(filePath,
FileMode.Open, FileAccess.Read, FileShare.Read,
bufferSize: 4096, useAsync: true))
{
StringBuilder sb = new StringBuilder();
byte[] buffer = new byte[0x1000];
int numRead;
while ((numRead = await sourceStream.ReadAsync(buffer, 0, buffer.Length)) != 0)
{
string text = Encoding.Unicode.GetString(buffer, 0, numRead);
sb.Append(text);
}
return sb.ToString();
}
}
Right, so it turns out that my problem was using a textbox for writing the string to instead of a richtextbox. This fixed my problem. Thanks for your answers.
I handle with big files(of which capacities is minimum 500MB) to split and merge by c#.
I have to split the file into thousands of files, sort these files into some groups, and merge these by each group.
The minimum number of files are 10,000.
I implement the merge function by using the method Stream.CopyTo(). Here is the main part of that.
using (Stream writer = File.OpenWrite(outputFilePath))
{
int fileNum = filePaths.Count();
for (int i = 0; i < fileNum; i++)
{
using (Stream reader = File.OpenRead(filePaths.ElementAt(i)))
{ reader.CopyTo(writer); }
}
}
I've tested my program to split 500MB into 17000 files of 2 groups and to merge each group of 8500 files into one file.
The merging part takes about 80 seconds. I think it is pretty slow compared to splitting the same file which takes about 15~20 seconds
Is there any method which is faster than my code?
Your code looks fine but ElementAt is a code smell. Convert that to an array and use [i] instead. If you have 10K elements I'm positive you're wasting a lot of time.
Why not just use the Stream.CopyTo() method?
private static void CombineMultipleFilesIntoSingleFile(string inputDirectoryPath, string inputFileNamePattern, string outputFilePath)
{
string[] inputFilePaths = Directory.GetFiles(inputDirectoryPath, inputFileNamePattern);
Console.WriteLine("Number of files: {0}.", inputFilePaths.Length);
using (var outputStream = File.Create(outputFilePath))
{
foreach (var inputFilePath in inputFilePaths)
{
using (var inputStream = File.OpenRead(inputFilePath))
{
// Buffer size can be passed as the second argument.
inputStream.CopyTo(outputStream);
}
Console.WriteLine("The file {0} has been processed.", inputFilePath);
}
}
}
OR
Do it in chunks:
const int chunkSize = 2 * 1024; // 2KB
var inputFiles = new[] ;
using (var output = File.Create("output.dat"))
{
foreach (var file in inputFiles)
{
using (var input = File.OpenRead(file))
{
var buffer = new byte[chunkSize];
int bytesRead;
while ((bytesRead = input.Read(buffer, 0, buffer.Length)) > 0)
{
output.Write(buffer, 0, bytesRead);
}
}
}
}
Maybe try compressing the files?
using System;
using System.Collections.Generic;
using System.Text;
using System.IO;
using System.IO.Compression;
class Program {
static void SaveCompressedFile(string filename, string data) {
FileStream fileStream = new FileStream(filename, FileMode.Create, FileAccess.Write);
GZipStream compressionStream = new GZipStream(fileStream, CompressionMode.Compress);
StreamWriter writer = new StreamWriter(compressionStream);
writer.Write(data);
writer.Close();
}
static string LoadCompressedFile(string filename) {
FileStream fileStream = new FileStream(filename, FileMode.Open, FileAccess.Read);
GZipStream compressionStream = new GZipStream(fileStream, CompressionMode.Decompress);
StreamReader reader = new StreamReader(compressionStream);
string data = reader.ReadToEnd();
reader.Close();
return data;
}
static void Main(string[] args) {
try {
string filename = "compressedFile.txt";
string sourceString = "Source String";
SaveCompressedFile(filename, sourceString);
FileInfo compressedFileData = new FileInfo(filename);
string recoveredString = LoadCompressedFile(filename);
} catch (IOException ex) {
Console.WriteLine(ex.ToString());
}
}
}
Source
Also check out the example of compressing a directory.
using System;
using System.Text;
using System.IO;
using System.IO.Compression;
namespace CmprDir
{
class Program
{
delegate void ProgressDelegate(string sMessage);
static void CompressFile(string sDir, string sRelativePath, GZipStream zipStream)
{
//Compress file name
char[] chars = sRelativePath.ToCharArray();
zipStream.Write(BitConverter.GetBytes(chars.Length), 0, sizeof(int));
foreach (char c in chars)
zipStream.Write(BitConverter.GetBytes(c), 0, sizeof(char));
//Compress file content
byte[] bytes = File.ReadAllBytes(Path.Combine(sDir, sRelativePath));
zipStream.Write(BitConverter.GetBytes(bytes.Length), 0, sizeof(int));
zipStream.Write(bytes, 0, bytes.Length);
}
static bool DecompressFile(string sDir, GZipStream zipStream, ProgressDelegate progress)
{
//Decompress file name
byte[] bytes = new byte[sizeof(int)];
int Readed = zipStream.Read(bytes, 0, sizeof(int));
if (Readed < sizeof(int))
return false;
int iNameLen = BitConverter.ToInt32(bytes, 0);
bytes = new byte[sizeof(char)];
StringBuilder sb = new StringBuilder();
for (int i = 0; i < iNameLen; i++)
{
zipStream.Read(bytes, 0, sizeof(char));
char c = BitConverter.ToChar(bytes, 0);
sb.Append(c);
}
string sFileName = sb.ToString();
if (progress != null)
progress(sFileName);
//Decompress file content
bytes = new byte[sizeof(int)];
zipStream.Read(bytes, 0, sizeof(int));
int iFileLen = BitConverter.ToInt32(bytes, 0);
bytes = new byte[iFileLen];
zipStream.Read(bytes, 0, bytes.Length);
string sFilePath = Path.Combine(sDir, sFileName);
string sFinalDir = Path.GetDirectoryName(sFilePath);
if (!Directory.Exists(sFinalDir))
Directory.CreateDirectory(sFinalDir);
using (FileStream outFile = new FileStream(sFilePath, FileMode.Create, FileAccess.Write, FileShare.None))
outFile.Write(bytes, 0, iFileLen);
return true;
}
static void CompressDirectory(string sInDir, string sOutFile, ProgressDelegate progress)
{
string[] sFiles = Directory.GetFiles(sInDir, "*.*", SearchOption.AllDirectories);
int iDirLen = sInDir[sInDir.Length - 1] == Path.DirectorySeparatorChar ? sInDir.Length : sInDir.Length + 1;
using (FileStream outFile = new FileStream(sOutFile, FileMode.Create, FileAccess.Write, FileShare.None))
using (GZipStream str = new GZipStream(outFile, CompressionMode.Compress))
foreach (string sFilePath in sFiles)
{
string sRelativePath = sFilePath.Substring(iDirLen);
if (progress != null)
progress(sRelativePath);
CompressFile(sInDir, sRelativePath, str);
}
}
static void DecompressToDirectory(string sCompressedFile, string sDir, ProgressDelegate progress)
{
using (FileStream inFile = new FileStream(sCompressedFile, FileMode.Open, FileAccess.Read, FileShare.None))
using (GZipStream zipStream = new GZipStream(inFile, CompressionMode.Decompress, true))
while (DecompressFile(sDir, zipStream, progress));
}
public static int Main(string[] argv)
{
if (argv.Length != 2)
{
Console.WriteLine("Usage: CmprDir.exe <in_dir compressed_file> | <compressed_file out_dir>");
return 1;
}
string sDir;
string sCompressedFile;
bool bCompress = false;
try
{
if (Directory.Exists(argv[0]))
{
sDir = argv[0];
sCompressedFile = argv[1];
bCompress = true;
}
else
if (File.Exists(argv[0]))
{
sCompressedFile = argv[0];
sDir = argv[1];
bCompress = false;
}
else
{
Console.Error.WriteLine("Wrong arguments");
return 1;
}
if (bCompress)
CompressDirectory(sDir, sCompressedFile, (fileName) => { Console.WriteLine("Compressing {0}...", fileName); });
else
DecompressToDirectory(sCompressedFile, sDir, (fileName) => { Console.WriteLine("Decompressing {0}...", fileName); });
return 0;
}
catch (Exception ex)
{
Console.Error.WriteLine(ex.Message);
return 1;
}
}
}
}
Source
I try to convert zip file to byte[] and write it to a text file.
int BufferSize=65536;
private void button1_Click(object sender, EventArgs e)
{
DialogResult re = openFileDialog1.ShowDialog();
if (re == DialogResult.OK)
{
string fileName = openFileDialog1.FileName;
try
{
byte[] bytes = File.ReadAllBytes(fileName);
File.WriteAllBytes(#"F:\Info.txt", bytes);
}
catch (Exception) { }
}
}
Then I try to convert those byte to zip file. But I can't do it.
My code is here:
private void button2_Click(object sender, EventArgs e)
{
DialogResult re = openFileDialog1.ShowDialog();
if (re == DialogResult.OK)
{
string fileName = openFileDialog1.FileName;
try
{
byte[] bytes = File.ReadAllBytes(fileName);
using (var mstrim = new MemoryStream(bytes))
{
using (var inStream = new GZipStream(mstrim, CompressionMode.Compress))
{
using (var outStream = File.Create("Tax.Zip"))
{
var buffer = new byte[BufferSize];
int readBytes;
while ((readBytes = inStream.Read(buffer, 0, BufferSize)) != 0)
{
outStream.Write(buffer, 0, readBytes);
}
}
}
}
}
catch (Exception) { }
}
}
Error:File Mode not valid.
What File Mode is needed and how can I accomplish what I described?
Just try this.
byte[] data = File.ReadAllBytes("D:\\z.7z");
File.WriteAllBytes("D:\\t.txt", data); // Requires System.IO
byte[] newdata = File.ReadAllBytes("D:\\t.txt");
File.WriteAllBytes("D:\\a.7z", newdata); // Requires System.IO
Try this,
private void button1_Click(object sender, EventArgs e)
{
byte[] arr;
MemoryStream ms = new MemoryStream();
arr = File.ReadAllBytes("C:\\asik.zip");
File.WriteAllBytes(#"D:\\asik.txt", arr);
ms.Close();
FileStream stream = File.OpenRead(#"D:\\asik.txt");
byte[] fileBytes = new byte[stream.Length];
stream.Read(fileBytes, 0, fileBytes.Length);
stream.Close();
MemoryStream ms1 = new MemoryStream(fileBytes);
CreateToMemoryStream(ms1, #"D:\\asik.zip");
ms1.Close();
}
public void CreateToMemoryStream(MemoryStream memStreamIn, string zipEntryName)
{
MemoryStream outputMemStream = new MemoryStream();
ZipOutputStream zipStream = new ZipOutputStream(outputMemStream);
zipStream.SetLevel(3); //0-9, 9 being the highest level of compression
ZipEntry newEntry = new ZipEntry(zipEntryName);
newEntry.DateTime = DateTime.Now;
zipStream.PutNextEntry(newEntry);
StreamUtils.Copy(memStreamIn, zipStream, new byte[4096]);
zipStream.CloseEntry();
zipStream.IsStreamOwner = false; // False stops the Close also Closing the underlying stream.
zipStream.Close(); // Must finish the ZipOutputStream before using outputMemStream.
//outputMemStream.Position = 0;
//return outputMemStream;
//// Alternative outputs:
//// ToArray is the cleaner and easiest to use correctly with the penalty of duplicating allocated memory.
//byte[] byteArrayOut = outputMemStream.ToArray();
//// GetBuffer returns a raw buffer raw and so you need to account for the true length yourself.
//byte[] byteArrayOut2 = outputMemStream.GetBuffer();
//long len = outputMemStream.Length;
}
You're using GZipStream, which is used for GZip files, not (PK-)Zip files. This isn't going to work, obviously. Try the ZipFile class instead (though sadly, it doesn't work on streams, just files).
Apart from simply being a different file format, the big difference is that GZip is for compression only, while Zip is also an archive (that is, it can contain multiple files).
public class BytesVal {
public static void main(String[] args) throws IOException, MoreZipException {
// TODO Auto-generated method stub
File file = new File("F:\\ssd\\doc\\");
System.out.println("Byte inside the Zip file is" + BytesVal.getAllBytes(file));
}
public static byte[] getAllBytes(File folderName) throws IOException, MoreZipException {
String[] sourceFiles = null;
if (folderName.isDirectory()) {
sourceFiles = folderName.list();
if (sourceFiles.length > 1) {
throw new MoreZipException(sourceFiles.length);
}
}
byte[] bytes = null;
Path filePath = Paths.get("F:/ssd/doc/" + sourceFiles[0]);
bytes = Files.readAllBytes(filePath);
return bytes;
}
}
I am attempting to search files for specific text using an async Task. The output I receive is in chinese characters instead of what is actually in the text file I am parsing. Here is the code:
public async void ProcessRead(string fileLocation)
{
string filePath = fileLocation;
if (File.Exists(filePath) == false)
{
Debug.WriteLine("file not found: " + filePath);
}
else
{
try
{
string text = await ReadTextAsync(filePath);
Debug.WriteLine(text);
if (text.Contains(inputContentPattern))
{
addToDataGrid(Path.GetFileName(filePath), filePath, "");
}
}
catch (Exception ex)
{
Debug.WriteLine(ex.Message);
}
}
findButton.Enabled = true;
cancelFindButton.Enabled = false;
}
private async Task<string> ReadTextAsync(string filePath)
{
using (FileStream sourceStream = new FileStream(filePath,
FileMode.Open, FileAccess.Read, FileShare.Read,
bufferSize: 4096, useAsync: true))
{
StringBuilder sb = new StringBuilder();
byte[] buffer = new byte[0x1000];
int numRead;
while ((numRead = await sourceStream.ReadAsync(buffer, 0, buffer.Length)) != 0)
{
string text = Encoding.Unicode.GetString(buffer, 0, numRead);
sb.Append(text);
}
return sb.ToString();
}
}
You are reading the file using Unicode encoding (in an incorrect way). Find out what encoding the file actually has and use that encoding. Simply use File.ReadAllText and delete ReadTextAsync.
Async IO brings you zero performance benefits here so just don't do it. If this is a UI app use await Task.Run(() => File.ReadAllText(...)).