I'm not sure where and what am I doing wrong, but the zip that I'm creating using DotNetZip library, is creating a zip file whose contents are blank. Or the size of file in zip is showing as 0Kb and unable to open it.
Code:
public static async Task DotNetZipFileAsync(MemoryStream stream, string bucket, List<List<string>> pdfFileSet, IAmazonS3 s3Client)
{
using Ionic.Zip.ZipFile zip = new ZipFile();
foreach (var pdfFile in pdfFileSet)
{
foreach (var file in pdfFile)
{
GetObjectRequest request = new GetObjectRequest
{
BucketName = bucket,
Key = file
};
using GetObjectResponse response = await s3Client.GetObjectAsync(request);
using Stream responseStream = response.ResponseStream;
ZipEntry zipEntry = zip.AddEntry(file.Split('/')[^1], responseStream);
await responseStream.CopyToAsync(stream);
}
}
zip.Save(stream);
stream.Seek(0,SeekOrigin.Begin);
await stream.CopyToAsync(new FileStream(#"C:\LocalRepo\Temp.zip", FileMode.OpenOrCreate, FileAccess.ReadWrite, FileShare.ReadWrite));
}
}
}
Your code has at least two problems:
The read stream is completely consumed by the await responseStream.CopyToAsync(stream). You could rewind the responseStream to cope with this, but saving the data into the memory stream is completely useless.
The response stream is disposed before zip.Save is called.
What you could do: keep the streams open until Save is called and dispose them afterwards. As Alexey Rumyantsev discovered (see comments), also the GetObjectResponse objects need to be kept until the ZIP file is saved.
using Ionic.Zip.ZipFile zip = new ZipFile();
var disposables = List<IDisposable>();
try
{
foreach (var pdfFile in pdfFileSet)
{
foreach (var file in pdfFile)
{
GetObjectRequest request = new GetObjectRequest
{
BucketName = bucket,
Key = file
};
var response = await s3Client.GetObjectAsync(request);
disposables.Add(response);
var responseStream = response.ResponseStream;
disposables.Add(responseStream);
ZipEntry zipEntry = zip.AddEntry(file.Split('/')[^1], responseStream);
}
}
using var fileStream = new FileStream(#"C:\LocalRepo\Temp.zip", FileMode.Create, FileAccess.Write);
zip.Save(fileStream);
}
finally
{
foreach (var disposable in disposables)
{
disposable.Dispose();
}
}
The documentation has some hints ony how this could be made smarter.
public static async Task DotNetZipFileAsync(string bucket, List<List<string>> pdfFileSet, IAmazonS3 s3Client)
{
int read;
using Ionic.Zip.ZipFile zip = new ZipFile();
byte[] buffer = new byte[16 * 1024];
foreach (var pdfFile in pdfFileSet)
{
foreach (var file in pdfFile)
{
GetObjectRequest request = new GetObjectRequest
{
BucketName = bucket,
Key = file
};
using GetObjectResponse response = await s3Client.GetObjectAsync(request);
using Stream responseStream = response.ResponseStream;
using (MemoryStream ms = new MemoryStream())
{
while ((read = responseStream.Read(buffer, 0, buffer.Length)) > 0)
{
ms.Write(buffer, 0, read);
}
zip.AddEntry(file.Split('/')[^1], ms.ToArray());
}
}
}
using var fileStream = new FileStream(#"C:\LocalRepo\Temp.zip", FileMode.Create, FileAccess.Write);
zip.Save(fileStream);
}
Related
I am trying to upload file to amazon s3 but got error Cannot Access a closed stream in await client.PutObjectAsync(request);
using (var stream = new MemoryStream())
{
using (var sWriter = new StreamWriter(stream, Encoding.UTF8))
{
await sWriter.WriteAsync(commandWithMetadata.SerializeToString());
stream.Seek(0, SeekOrigin.Begin);
var fileName = GetFileName(command);
var request = new PutObjectRequest
{
BucketName = BucketName,
Key = fileName,
InputStream = stream
};
await client.PutObjectAsync(request);
}
}
there is AutoCloseStream property on request which by default is true and amazon lib is closing the stream automatically
I have the following code:
private static byte[] ConverterStringToByte(Stream body)
{
string fileName = "data_" + DateTime.Now.ToString("yyyyMMddhhmmss") + ".zip";
// Take out the bytes from the memory stream and safely close the stream
using (var ms = new MemoryStream())
{
body.CopyTo(ms);
using (var zipArchive = new ZipArchive(ms, ZipArchiveMode.Create, false))
{
var zipEntry = zipArchive.CreateEntry(fileName, CompressionLevel.Optimal);
using (BinaryWriter writer = new BinaryWriter(zipEntry.Open()))
{
ms.Position = 0;
writer.Write(ms.ToArray());
}
}
return ms.ToArray();
}
}
I am downloading the file successfully, however I'm getting
invalid file
when trying to open
I think it should be something like this. Not sure about fileName though, because it's the name of the file being put into archive, so I don't think it should have *.zip extension. Unless you are creating a zip of zips.
static byte[] ConverterStringToByte(Stream body)
{
string fileName = #"data_" + DateTime.Now.ToString("yyyyMMddhhmmss") + ".zip";
using (var ms = new MemoryStream())
{
using (var zipArchive = new ZipArchive(ms, ZipArchiveMode.Create, false))
{
var zipEntry = zipArchive.CreateEntry(fileName, CompressionLevel.Optimal);
using (var destStream = zipEntry.Open())
{
body.CopyTo(destStream);
}
}
return ms.ToArray();
}
}
I have a DLL with embedded Excel file. The goal is to retrieve this file and create some entry (Empty_File.txt in this example). When I'm using FileStream - the entry gets created, but when I'm using MemoryStream - entry isn't created.
var filePath = "C:\\Temp\\Test2.xlsx";
var asm = typeof(Program).Assembly;
var asmName = asm.GetName().Name;
using var resourceStream = asm.GetManifestResourceStream($"{asmName}.Resources.Template.xlsx");
if (File.Exists(filePath)) File.Delete(filePath);
await UseFileStream(resourceStream, filePath);
// or
await UseMemoryStream(resourceStream, filePath);
static async Task UseMemoryStream(Stream resourceStream, string filePath)
{
using (var ms = new MemoryStream())
{
await resourceStream.CopyToAsync(ms);
using (var zip = new ZipArchive(ms, ZipArchiveMode.Update))
{
zip.CreateEntry("Empty_File.txt");
using (var fs = CreateFileStream(filePath))
{
ms.Seek(0L, SeekOrigin.Begin);
await ms.CopyToAsync(fs);
}
}
}
}
static async Task UseFileStream(Stream resourceStream, string filePath)
{
using var fs = CreateFileStream(filePath);
await resourceStream.CopyToAsync(fs);
using var zip = new ZipArchive(fs, ZipArchiveMode.Update);
zip.CreateEntry("Empty_File.txt");
}
static FileStream CreateFileStream(string filePath) =>
new FileStream(filePath, new FileStreamOptions
{
Access = FileAccess.ReadWrite,
Mode = FileMode.Create,
Share = FileShare.None
});
Per the docs for ZipArchive.Dispose:
This method finishes writing the archive and releases all resources used by the ZipArchive object. Unless you construct the object by using the ZipArchive(Stream, ZipArchiveMode, Boolean) constructor overload and set its leaveOpen parameter to true, all underlying streams are closed and no longer available for subsequent write operations.
You are currently writing to the file stream before this happens, so the changes to the zip file haven't been written yet.
You'll also note from this that the underlying MemoryStream will be disposed unless you specify leaveOpen: true in the constructor, which would prevent you copying to the file afterwards.
So putting both of these together:
static async Task UseMemoryStream(Stream resourceStream, string filePath)
{
using (var ms = new MemoryStream())
{
await resourceStream.CopyToAsync(ms);
using (var zip = new ZipArchive(ms, ZipArchiveMode.Update, leaveOpen: true))
{
zip.CreateEntry("Empty_File.txt");
}
using (var fs = CreateFileStream(filePath))
{
ms.Seek(0L, SeekOrigin.Begin);
await ms.CopyToAsync(fs);
}
}
}
I am creating a zip file that, appears, valid but is always invalid after I have put it to a Amazon S3 bucket. I am using System.IO.Compression for the task and AmazonS3Client for uploading:
private byte[] GenerateZipFile(string tenant)
{
byte[] zipData;
var results = QueryAggregateTable(tenant);
using (var memoryStream = new MemoryStream())
using (var archive = new ZipArchive(memoryStream, ZipArchiveMode.Create, true))
{
foreach (var item in results)
{
var archiveEntry = archive.CreateEntry($"{item.RowKey:D3}.json", CompressionLevel.Fastest);
using (var entryStream = archiveEntry.Open())
{
var entryBytes = Encoding.UTF8.GetBytes(item.Data);
entryStream.Write(entryBytes, 0, item.Data.Length);
}
}
zipData = memoryStream.ToArray();
}
return zipData;
}
and
private async Task UploadToAmazon(byte[] zipData, string tenant)
{
var bucketName = _config["egestionBucketName"];
var configCreds = _config["egestionAwsCredentials"].Split(":");
var awsCreds = new BasicAWSCredentials(configCreds[0], configCreds[1]);
var awsRegion = Amazon.RegionEndpoint.GetBySystemName(_config["egestionRegionEndpointSystemName"]);
var s3Client = new AmazonS3Client(awsCreds, awsRegion);
using (var stream = new MemoryStream(zipData))
{
var putRequest = new PutObjectRequest
{
BucketName = bucketName,
Key = $"{tenant}-{DateTime.UtcNow.ToString("s")}.zip",
InputStream = stream,
CannedACL = S3CannedACL.BucketOwnerFullControl
};
await s3Client.PutObjectAsync(putRequest);
}
}
The byte array looks good after returning from generation and the upload method does, in fact, load a file with the correct name to the bucket. When I attempt to download the file to check it I cannot open it with a message stating it is invalid.
I have had some problems with async/await and suspect it may be something related but there is no non async option for PutObject that I can find. Any help appreciated.
This is not an async-await issue.
The bytes from the memory stream are being collected before the archive has had a chance to write all the data to the stream. The uploaded archive is incomplete and therefore invalid when downloaded.
Move
zipData = memoryStream.ToArray();
to outside of the archive using block so that any buffered data is flushed to the backing stream when the archive is disposed.
//...
using (var memoryStream = new MemoryStream()) {
using (var archive = new ZipArchive(memoryStream, ZipArchiveMode.Create, true)) {
foreach (var item in results) {
var archiveEntry = archive.CreateEntry($"{item.RowKey:D3}.json", CompressionLevel.Fastest);
using (var entryStream = archiveEntry.Open()) {
var entryBytes = Encoding.UTF8.GetBytes(item.Data);
entryStream.Write(entryBytes, 0, entryBytes.Length);
}
}
}//Archive disposed and pushed any remaining buffered data to the stream.
zipData = memoryStream.ToArray();
}
//...
I'm retrieving a file from Amazon S3. I want to convert the file to bytes so that I can download it as follows:
var download = new FileContentResult(bytes, "application/pdf");
download.FileDownloadName = filename;
return download;
I have the file here:
var client = Amazon.AWSClientFactory.CreateAmazonS3Client(
accessKey,
secretKey,
config
);
GetObjectRequest request = new GetObjectRequest();
GetObjectResponse response = client.GetObject(request);
I know about response.WriteResponseStreamToFile() but I want to download the file to the regular downloads folder. If I convert the GetObjectResponse to bytes, I can return the file. How can I do this?
Here's the solution I found for anyone else who needs it:
GetObjectResponse response = client.GetObject(request);
using (Stream responseStream = response.ResponseStream)
{
var bytes = ReadStream(responseStream);
var download = new FileContentResult(bytes, "application/pdf");
download.FileDownloadName = filename;
return download;
}
public static byte[] ReadStream(Stream responseStream)
{
byte[] buffer = new byte[16 * 1024];
using (MemoryStream ms = new MemoryStream())
{
int read;
while ((read = responseStream.Read(buffer, 0, buffer.Length)) > 0)
{
ms.Write(buffer, 0, read);
}
return ms.ToArray();
}
}
Just another option:
Stream rs;
using (IAmazonS3 client = Amazon.AWSClientFactory.CreateAmazonS3Client())
{
GetObjectRequest getObjectRequest = new GetObjectRequest();
getObjectRequest.BucketName = "mybucketname";
getObjectRequest.Key = "mykey";
using (var getObjectResponse = client.GetObject(getObjectRequest))
{
getObjectResponse.ResponseStream.CopyTo(rs);
}
}
I struggled to get the cleaner method offered by Alex to work (not sure what I'm missing), but I wanted to do it w/o the extra ReadStream method offered by Erica (although it worked)... here is what I wound up doing:
var s3Client = new AmazonS3Client(AccessKeyId, SecretKey, Amazon.RegionEndpoint.USEast1);
using (s3Client)
{
MemoryStream ms = new MemoryStream();
GetObjectRequest getObjectRequest = new GetObjectRequest();
getObjectRequest.BucketName = BucketName;
getObjectRequest.Key = awsFileKey;
using (var getObjectResponse = s3Client.GetObject(getObjectRequest))
{
getObjectResponse.ResponseStream.CopyTo(ms);
}
var download = new FileContentResult(ms.ToArray(), "image/png"); //"application/pdf"
download.FileDownloadName = ToFilePath;
return download;
}
Stream now has asynchronous methods. In C# 8, you can do this:
public async Task<byte[]> GetAttachmentAsync(string objectPointer)
{
var objReq = new GetObjectRequest
{
BucketName = "bucket-name",
Key = objectPointer, // the file name
};
using var objResp = await _s3Client.GetObjectAsync(objReq);
using var ms = new MemoryStream();
await objResp.ResponseStream.CopyToAsync(ms, _ct); // _ct is a CancellationToken
return ms.ToArray();
}
This won't block any threads while the IO occurs.