I am trying to upload file to amazon s3 but got error Cannot Access a closed stream in await client.PutObjectAsync(request);
using (var stream = new MemoryStream())
{
using (var sWriter = new StreamWriter(stream, Encoding.UTF8))
{
await sWriter.WriteAsync(commandWithMetadata.SerializeToString());
stream.Seek(0, SeekOrigin.Begin);
var fileName = GetFileName(command);
var request = new PutObjectRequest
{
BucketName = BucketName,
Key = fileName,
InputStream = stream
};
await client.PutObjectAsync(request);
}
}
there is AutoCloseStream property on request which by default is true and amazon lib is closing the stream automatically
Related
When I post the below code using httpclient
using var formContent = new MultipartFormDataContent("NKdKd9Yk");
using var stream = new MemoryStream();
file.CopyTo(stream);
var fileBytes = stream.ToArray();
formContent.Headers.ContentType.MediaType = "multipart/form-data";
formContent.Add(new StreamContent(stream), "file", fileName);
var response = await httpClient.PostAsync(GetDocumentUpdateRelativeUrl(), formContent);
here file is of type IFormfile
In API side I retrieve file as follows
var base64str= "";
using (var ms = new MemoryStream())
{
request.file.CopyTo(ms);
var fileBytes = ms.ToArray();
base64str= Convert.ToBase64String(fileBytes);
// act on the Base64 data
}
I get 0 byte. My questions is what's wrong with this approch?
But If I use below code. Then API works and I get what I post.
using var formContent = new MultipartFormDataContent("NKdKd9Yk");
using var stream = new MemoryStream();
file.CopyTo(stream);
var fileBytes = stream.ToArray();
formContent.Headers.ContentType.MediaType = "multipart/form-data";
formContent.Add(new StreamContent(new MemoryStream(fileBytes)), "file", fileName);
differece is how I add stream content
formContent.Add(new StreamContent(stream), "file", fileName);
vs
formContent.Add(new StreamContent(new MemoryStream(fileBytes)), "file", fileName);
Why the first approch didn't work but second one does?
You need to add stream.Seek(0, SeekOrigin.Begin); in order to jump back to the beginning of the MemoryStream. You should also use CopyToAsync
In the second version, you had a fresh MemoryStream from the byte[] array, which is positioned on 0 anyway.
using var formContent = new MultipartFormDataContent("NKdKd9Yk");
using var stream = new MemoryStream();
await file.CopyToAsync(stream);
stream.Seek(0, SeekOrigin.Begin);
formContent.Headers.ContentType.MediaType = "multipart/form-data";
formContent.Add(new StreamContent(stream), "file", fileName);
using var response = await httpClient.PostAsync(GetDocumentUpdateRelativeUrl(), formContent);
Although to be honest, the MemoryStream seems entirely unnecessary here. Just pass the a Stream from file directly.
using var formContent = new MultipartFormDataContent("NKdKd9Yk");
formContent.Headers.ContentType.MediaType = "multipart/form-data";
using var stream = file.OpenReadStream();
formContent.Add(new StreamContent(stream), "file", fileName);
using var response = await httpClient.PostAsync(GetDocumentUpdateRelativeUrl(), formContent);
I'm not sure where and what am I doing wrong, but the zip that I'm creating using DotNetZip library, is creating a zip file whose contents are blank. Or the size of file in zip is showing as 0Kb and unable to open it.
Code:
public static async Task DotNetZipFileAsync(MemoryStream stream, string bucket, List<List<string>> pdfFileSet, IAmazonS3 s3Client)
{
using Ionic.Zip.ZipFile zip = new ZipFile();
foreach (var pdfFile in pdfFileSet)
{
foreach (var file in pdfFile)
{
GetObjectRequest request = new GetObjectRequest
{
BucketName = bucket,
Key = file
};
using GetObjectResponse response = await s3Client.GetObjectAsync(request);
using Stream responseStream = response.ResponseStream;
ZipEntry zipEntry = zip.AddEntry(file.Split('/')[^1], responseStream);
await responseStream.CopyToAsync(stream);
}
}
zip.Save(stream);
stream.Seek(0,SeekOrigin.Begin);
await stream.CopyToAsync(new FileStream(#"C:\LocalRepo\Temp.zip", FileMode.OpenOrCreate, FileAccess.ReadWrite, FileShare.ReadWrite));
}
}
}
Your code has at least two problems:
The read stream is completely consumed by the await responseStream.CopyToAsync(stream). You could rewind the responseStream to cope with this, but saving the data into the memory stream is completely useless.
The response stream is disposed before zip.Save is called.
What you could do: keep the streams open until Save is called and dispose them afterwards. As Alexey Rumyantsev discovered (see comments), also the GetObjectResponse objects need to be kept until the ZIP file is saved.
using Ionic.Zip.ZipFile zip = new ZipFile();
var disposables = List<IDisposable>();
try
{
foreach (var pdfFile in pdfFileSet)
{
foreach (var file in pdfFile)
{
GetObjectRequest request = new GetObjectRequest
{
BucketName = bucket,
Key = file
};
var response = await s3Client.GetObjectAsync(request);
disposables.Add(response);
var responseStream = response.ResponseStream;
disposables.Add(responseStream);
ZipEntry zipEntry = zip.AddEntry(file.Split('/')[^1], responseStream);
}
}
using var fileStream = new FileStream(#"C:\LocalRepo\Temp.zip", FileMode.Create, FileAccess.Write);
zip.Save(fileStream);
}
finally
{
foreach (var disposable in disposables)
{
disposable.Dispose();
}
}
The documentation has some hints ony how this could be made smarter.
public static async Task DotNetZipFileAsync(string bucket, List<List<string>> pdfFileSet, IAmazonS3 s3Client)
{
int read;
using Ionic.Zip.ZipFile zip = new ZipFile();
byte[] buffer = new byte[16 * 1024];
foreach (var pdfFile in pdfFileSet)
{
foreach (var file in pdfFile)
{
GetObjectRequest request = new GetObjectRequest
{
BucketName = bucket,
Key = file
};
using GetObjectResponse response = await s3Client.GetObjectAsync(request);
using Stream responseStream = response.ResponseStream;
using (MemoryStream ms = new MemoryStream())
{
while ((read = responseStream.Read(buffer, 0, buffer.Length)) > 0)
{
ms.Write(buffer, 0, read);
}
zip.AddEntry(file.Split('/')[^1], ms.ToArray());
}
}
}
using var fileStream = new FileStream(#"C:\LocalRepo\Temp.zip", FileMode.Create, FileAccess.Write);
zip.Save(fileStream);
}
I have written a Post method in ASP.NET Core to compress the requests body and upload it to Azure Blob Storage. The method takes parameters as follows:
public async Task<IActionResult> Post([FromHeader] string AssignmentId)
Various strings are then set, including fetching the connection string for the storage:
string fileName = $"{AssignmentId}.gz";
string compressedFilePath = Path.Combine(hostEnvironment.ContentRootPath, $"Test JSONs/{fileName}");
string connectionString = Environment.GetEnvironmentVariable("AZURE_STORAGE_CONNECTION_STRING");
I initialize the BlobClient:
BlobClient blobClient = new BlobClient(connectionString, "assignments", fileName);
Then I create a file, and compress the body stream of the request using GZipStream to the file:
using (FileStream compressedFileStream = System.IO.File.Create(compressedFilePath))
{
using GZipStream compressionStream = new GZipStream(compressedFileStream, CompressionMode.Compress);
using Stream bodyStream = HttpContext.Request.Body;
await bodyStream.CopyToAsync(compressionStream);
}
Finally I read the file I just wrote and upload using the FileStream:
using (FileStream fileStream = System.IO.File.OpenRead(compressedFilePath))
{
await blobClient.UploadAsync(fileStream);
}
This solution works, but I am concerned about the constant reading and writing of the file, in terms of speed. I attempted to use a MemoryStream passed into the GZipStream, however it ended up only uploading 10B files when the files should be 1KB+.
I appreciate any suggestions.
Here is the complete method:
public async Task<IActionResult> Post([FromHeader] string AssignmentId)
{
string fileName = $"{AssignmentId}.gz";
string compressedFilePath = Path.Combine(hostEnvironment.ContentRootPath, $"Test JSONs/{fileName}");
string connectionString = Environment.GetEnvironmentVariable("AZURE_STORAGE_CONNECTION_STRING");
BlobClient blobClient = new BlobClient(connectionString, "assignments", fileName);
using (FileStream compressedFileStream = System.IO.File.Create(compressedFilePath))
{
using GZipStream compressionStream = new GZipStream(compressedFileStream, CompressionMode.Compress);
using Stream bodyStream = HttpContext.Request.Body;
await bodyStream.CopyToAsync(compressionStream);
}
using (FileStream fileStream = System.IO.File.OpenRead(compressedFilePath))
{
await blobClient.UploadAsync(fileStream);
}
return Ok();
}
I eventually solved this by both leaving the compression stream open, and by resetting the position of the memory stream the compression stream is writing to (thanks to #MitchWheat !).
using MemoryStream memoryStream = new MemoryStream() ;
using (Stream bodyStream = HttpContext.Request.Body)
{
using (GZipStream compressionStream = new GZipStream(memoryStream,
CompressionMode.Compress, true))
{
await bodyStream.CopyToAsync(compressionStream);
}
}
memoryStream.Position = 0;
await blobClient.UploadAsync(memoryStream, overwrite: true);
The Azure Storage library offers a push-based stream to upload content that works well with compression.
const int PageSize = 1024 * 1024; // 1 MB
using var sourceFile = new FileStream(file.FullName, FileMode.Open, FileAccess.Read, FileShare.Read);
using var destinationStream = await blobClient.OpenWriteAsync(overwrite: true, new() { BufferSize = PageSize }, cancellationToken);
using var compressedContent = new GZipStream(destinationStream, CompressionMode.Compress, leaveOpen: true);
await sourceFile.CopyToAsync(compressedContent, cancellationToken);
I have a wcf service here to upload the files to Amazon s3 server. After the successful upload, I need to delete the file from my local path. But when I try to delete the file, got an error says The process cannot access the file.Because its being used by another process".Sharing below my code snippets.
var putRequest = new PutObjectRequest
{
BucketName = System.Configuration.ConfigurationManager.AppSettings["S3Bucket"]
.ToString(),
Key = keyName,
FilePath = path,
ContentType = "application/pdf"
};
client = new AmazonS3Client(bucketRegion);
PutObjectResponse response = await client.PutObjectAsync(putRequest);
putRequest = null;
client.Dispose();
File.Delete(path);
If anyone know about the issue, please update..
There might be a timing issue here, so you might want to try to close the stream explicitly.
Do note, I am not sure, if I am mistaken I'll remove this, but it was to long for a comment.
using (var fileStream = new File.OpenRead(path))
{
var putRequest = new PutObjectRequest
{
BucketName = System.Configuration.ConfigurationManager.AppSettings["S3Bucket"]
.ToString(),
Key = keyName,
InputStream = fileStream ,
ContentType = "application/pdf",
AutoCloseStream = false,
};
using (var c = new AmazonS3Client(bucketRegion))
{
PutObjectResponse response = await c.PutObjectAsync(putRequest);
}
} //filestream should be closed here, if not: call fileStream.Close()
File.Delete(path);
More info on the properties: https://docs.aws.amazon.com/sdkfornet1/latest/apidocs/html/T_Amazon_S3_Model_PutObjectRequest.htm
I am creating a zip file that, appears, valid but is always invalid after I have put it to a Amazon S3 bucket. I am using System.IO.Compression for the task and AmazonS3Client for uploading:
private byte[] GenerateZipFile(string tenant)
{
byte[] zipData;
var results = QueryAggregateTable(tenant);
using (var memoryStream = new MemoryStream())
using (var archive = new ZipArchive(memoryStream, ZipArchiveMode.Create, true))
{
foreach (var item in results)
{
var archiveEntry = archive.CreateEntry($"{item.RowKey:D3}.json", CompressionLevel.Fastest);
using (var entryStream = archiveEntry.Open())
{
var entryBytes = Encoding.UTF8.GetBytes(item.Data);
entryStream.Write(entryBytes, 0, item.Data.Length);
}
}
zipData = memoryStream.ToArray();
}
return zipData;
}
and
private async Task UploadToAmazon(byte[] zipData, string tenant)
{
var bucketName = _config["egestionBucketName"];
var configCreds = _config["egestionAwsCredentials"].Split(":");
var awsCreds = new BasicAWSCredentials(configCreds[0], configCreds[1]);
var awsRegion = Amazon.RegionEndpoint.GetBySystemName(_config["egestionRegionEndpointSystemName"]);
var s3Client = new AmazonS3Client(awsCreds, awsRegion);
using (var stream = new MemoryStream(zipData))
{
var putRequest = new PutObjectRequest
{
BucketName = bucketName,
Key = $"{tenant}-{DateTime.UtcNow.ToString("s")}.zip",
InputStream = stream,
CannedACL = S3CannedACL.BucketOwnerFullControl
};
await s3Client.PutObjectAsync(putRequest);
}
}
The byte array looks good after returning from generation and the upload method does, in fact, load a file with the correct name to the bucket. When I attempt to download the file to check it I cannot open it with a message stating it is invalid.
I have had some problems with async/await and suspect it may be something related but there is no non async option for PutObject that I can find. Any help appreciated.
This is not an async-await issue.
The bytes from the memory stream are being collected before the archive has had a chance to write all the data to the stream. The uploaded archive is incomplete and therefore invalid when downloaded.
Move
zipData = memoryStream.ToArray();
to outside of the archive using block so that any buffered data is flushed to the backing stream when the archive is disposed.
//...
using (var memoryStream = new MemoryStream()) {
using (var archive = new ZipArchive(memoryStream, ZipArchiveMode.Create, true)) {
foreach (var item in results) {
var archiveEntry = archive.CreateEntry($"{item.RowKey:D3}.json", CompressionLevel.Fastest);
using (var entryStream = archiveEntry.Open()) {
var entryBytes = Encoding.UTF8.GetBytes(item.Data);
entryStream.Write(entryBytes, 0, entryBytes.Length);
}
}
}//Archive disposed and pushed any remaining buffered data to the stream.
zipData = memoryStream.ToArray();
}
//...