I'm following the documentation found here about doing multipart upload with the .NET client library.
The issue I'm having is that each part sent to S3 is overwriting the last part. So in other words my pieces are 10kb each (tried 5mb at a time too) and each upload overwrites the previous. What am I doing wrong?
Here's what I've got
var fileTransferUtility = new TransferUtility(_s3Client);
var request = new TransferUtilityUploadRequest
{
BucketName = "mybucket",
InputStream = stream,
StorageClass = S3StorageClass.ReducedRedundancy,
PartSize = stream.Length,//stream is 10,000 bytes at a time
Key = fileName
};
Edit
Here's working code for doing the multipart upload
public UploadPartResponse UploadChunk(Stream stream, string fileName, string uploadId, List<PartETag> eTags, int partNumber, bool lastPart)
{
stream.Position = 0;
//Step 1: build and send a multi upload request
if (partNumber == 1)
{
var initiateRequest = new InitiateMultipartUploadRequest
{
BucketName = _settings.Bucket,
Key = fileName
};
var initResponse = _s3Client.InitiateMultipartUpload(initiateRequest);
uploadId = initResponse.UploadId;
}
//Step 2: upload each chunk (this is run for every chunk unlike the other steps which are run once)
var uploadRequest = new UploadPartRequest
{
BucketName = _settings.Bucket,
Key = fileName,
UploadId = uploadId,
PartNumber = partNumber,
InputStream = stream,
IsLastPart = lastPart,
PartSize = stream.Length
};
var response = _s3Client.UploadPart(uploadRequest);
//Step 3: build and send the multipart complete request
if (lastPart)
{
eTags.Add(new PartETag
{
PartNumber = partNumber,
ETag = response.ETag
});
var completeRequest = new CompleteMultipartUploadRequest
{
BucketName = _settings.Bucket,
Key = fileName,
UploadId = uploadId,
PartETags = eTags
};
try
{
_s3Client.CompleteMultipartUpload(completeRequest);
}
catch
{
//do some logging and return null response
return null;
}
}
response.ResponseMetadata.Metadata["uploadid"] = uploadRequest.UploadId;
return response;
}
If you have a stream that is broken up into 10 chunks you will be hitting this method 10 times, on the first chunk you will hit step 1 & 2, chunks 2-9 only step 2 and on the last only step 3. Your need to send back to your client the upload id and the etag for each response. At step 3 you will need to provide the etag for all pieces or else it will put together the file on S3 without 1 more pieces. On my client side I had a hidden field where I persisted the etag list (comma delimited).
What this code sets up is a request that will upload an object with only one part because you pass in a stream and set the part size to the length of the whole stream.
The intention of using the TransferUtility is you would give it a large stream or file path and set part size to the increments you want the stream broke down to. You can also leave PartSize blank which will use a default part size.
Related
Use the OneDrive SDK to upload files.
At this time, you have to pass the file path, but uploading using the code takes a long time.
Can I upload files even if I pass the temporary file path?
Currently I get the file path after saving the file to the server.
In this case, an issue arises from speed problems.
Is there any way to look at the temporary file path?
public async Task<JObject> UploadLargeFiles(string upn, IFormFile files)
{
var jObject = new JObject();
int fileSize = Convert.ToInt32(files.Length);
var folderName = Path.Combine("wwwroot", "saveLargeFiles");
var pathToSave = Path.Combine(System.IO.Directory.GetCurrentDirectory(), folderName);
var fullPath = "";
if (files.Length > 0)
{
var fileName = files.FileName;
fullPath = Path.Combine(pathToSave, fileName);
using (var stream = new FileStream(fullPath, FileMode.Create))
files.CopyTo(stream);
}
var filePath = fullPath;
var fileStream = System.IO.File.OpenRead(filePath);
GraphServiceClient client = await MicrosoftGraphClient.GetGraphServiceClient();
var uploadProps = new DriveItemUploadableProperties
{
ODataType = null,
AdditionalData = new Dictionary<string, object>
{
{ "#microsoft.graph.conflictBehavior", "rename" }
}
};
var item = this.SelectUploadFolderID(upn).Result;
var uploadSession = await client.Users[upn].Drive.Items[item].ItemWithPath(files.FileName).CreateUploadSession(uploadProps).Request().PostAsync();
int maxChunkSize = 320 * 1024;
var uploadTask = new LargeFileUploadTask<DriveItem>(uploadSession, fileStream, maxChunkSize);
var response = await uploadTask.UploadAsync();
if (response.UploadSucceeded)
{
return
}
else
{
return null;
}
}
Your server's disk is probably not what makes this slow. By default, uploaded files are stored in a temporary directory, which you can save permanently by using CopyTo(FileStream) like you do.
You can skip this step and call IFormFile.OpenReadStream() to obtain a stream to the temporary file, then pass that to the LargeFileUploadTask.
Point is, it's probably the uploading to OneDrive that takes the largest amount of time. Depending on your setup, you may want to save files into a queue directory (the temp file gets deleted after the request completes), and have a background service read that queue and upload them to OneDrive.
My image are uploading locally but when i deployed lambda its giving a broken image(Note: it is uploading image but size increases),I have added Binary Media Type in the API Gateway , but still not getting right results. Interesting thing is that when i uploaded a text file it was perfect on the bucket but not images.
public async Task<S3Response> ImageUpload(IFormFile file ){
string bucket_name = "your_bucket";
var client = new AmazonS3Client("***", "****", RegionEndpoint.USEast1);
var stream = new System.IO.MemoryStream();
file.CopyTo(stream);
var request = new PutObjectRequest
{
Key = file.FileName,
BucketName = bucket_name,
InputStream = stream,
//ContentType = "application/octet-stream",
ContentType = file.ContentType,
CannedACL = S3CannedACL.PublicRead
};
response = await client.PutObjectAsync(request);
}
I am saving image as base64 string on s3 bucket and converting back from base64 string to my original image on the client side.If someone got a better solution kindly add in a thread.
byte[] byteArray = Encoding.UTF8.GetBytes(file.Filebase64);
stream= new MemoryStream(byteArray);
var request = new PutObjectRequest
{
Key=file.File_name,
BucketName = bucket_name,
InputStream = stream,
ContentType = "text/plain",
CannedACL = S3CannedACL.PublicRead
};
where Image file model class look like this :
public class ImageModel
{
public String File_name { set; get; }
public String Filebase64 { set; get; }
}
I'm using AWS Lambda C# .Net core
I am trying to upload a .jpg file without saving it to the local machine (not allowed in a deployed Lambda function)
I get the file in hex-string form and can re-code it into binary, save it as a file, and even upload it from my local debug normally.
int len = image.ImagePayload.Length;
byte[] bin = new byte[len / 2];
for (int i = 0; i < len; i += 2)
{
bin[i / 2] = Convert.ToByte(image.ImagePayload.Substring(i, 2), 16);
}
File.WriteAllBytes(image.ImageName, bin);
PutObjectRequest putObj = new PutObjectRequest
{
BucketName = input.Bucket,
FilePath = image.ImageName,
ContentType = "image/jpg",
Key = image.ImageName
};
PutObjectResponse putResp = S3Client.PutObjectAsync(putObj).Result;
AWS Lambda fails with "Read-only file system" when fully deployed
Any way to upload to S3 without saving the data to a file?
Instead of using FilePath, you can use InputStream on the PutObjectRequest.
byte bin = ...
using (var stream = new MemoryStream(bin))
{
var request = new PutObjectRequest
{
BucketName = input.Bucket,
InputStream = stream,
ContentType = "image/jpg",
Key = image.ImageName
}
var response = await S3Client.PutObjectAsync(request).ConfigureAwait(false);
}
Ref: https://docs.aws.amazon.com/sdkfornet1/latest/apidocs/html/T_Amazon_S3_Model_PutObjectRequest.htm
I am attempting to download a list of files from urls stored in my database, and then upload them to my Azure FileStorage account. I am successfully downloading the files and can turn them into files on my local storage or convert them to text and upload them. However I lose data when converting something like a pdf to a text and I do not want to have to store the files on the Azure app that this endpoint is hosted on as I do not need to manipulate the files in any way.
I have attempted to upload the files from the Stream I get from the HttpContent object using the UploadFromStream method on the CloudFile. Whenever this command is run I get an InvalidOperationException with the message "Operation is not valid due to the current state of the object."
I've tried converting the original Stream to a MemoryStream as well but this just writes a blank file to the FileStorage account, even if I set the position to the beginning of the MemoryStream. My code is below and if anyone could point out what information I am missing to make this work I would appreciate it.
public DownloadFileResponse DownloadFile(FileLink fileLink)
{
string fileName = string.Format("{0}{1}{2}", fileLink.ExpectedFileName, ".", fileLink.ExpectedFileType);
HttpStatusCode status;
string hash = "";
using (var client = new HttpClient())
{
client.Timeout = TimeSpan.FromSeconds(10); // candidate for .config setting
client.DefaultRequestHeaders.Add("User-Agent", USER_AGENT);
var request = new HttpRequestMessage(HttpMethod.Get, fileLink.ExpectedURL);
var sendTask = client.SendAsync(request, HttpCompletionOption.ResponseHeadersRead);
var response = sendTask.Result; // not ensuring success here, going to handle error codes without exceptions
status = response.StatusCode;
if (status == HttpStatusCode.OK)
{
var httpStream = response.Content.ReadAsStreamAsync().Result;
fileStorage.WriteFile(fileLink.ExpectedFileType, fileName, httpStream);
hash = HashGenerator.GetMD5HashFromStream(httpStream);
}
}
return new DownloadFileResponse(status, fileName, hash);
}
public void WriteFile(string targetDirectory, string targetFilePath, Stream fileStream)
{
var options = SetOptions();
var newFile = GetTargetCloudFile(targetDirectory, targetFilePath);
newFile.UploadFromStream(fileStream, options: options);
}
public FileRequestOptions SetOptions()
{
FileRequestOptions options = new FileRequestOptions();
options.ServerTimeout = TimeSpan.FromSeconds(10);
options.RetryPolicy = new NoRetry();
return options;
}
public CloudFile GetTargetCloudFile(string targetDirectory, string targetFilePath)
{
if (!shareConnector.share.Exists())
{
throw new Exception("Cannot access Azure File Storage share");
}
CloudFileDirectory rootDirectory = shareConnector.share.GetRootDirectoryReference();
CloudFileDirectory directory = rootDirectory.GetDirectoryReference(targetDirectory);
if (!directory.Exists())
{
throw new Exception("Target Directory does not exist");
}
CloudFile newFile = directory.GetFileReference(targetFilePath);
return newFile;
}
Had the same problem, the only way it worked is by reading the coming stream (in your case it is httpStream in DownloadFile(FileLink fileLink) method) to a byte array and using UploadFromByteArray (byte[] buffer, int index, int count) instead of UploadFromStream
So your WriteFile(FileLink fileLink) method will look like:
public void WriteFile(string targetDirectory, string targetFilePath, Stream fileStream)
{
var options = SetOptions();
var newFile = GetTargetCloudFile(targetDirectory, targetFilePath);
const int bufferLength= 600;
byte[] buffer = new byte[bufferLength];
// Buffer to read from stram This size is just an example
List<byte> byteArrayFile = new List<byte>(); // all your file will be here
int count = 0;
try
{
while ((count = fileStream.Read(buffer, 0, bufferLength)) > 0)
{
byteArrayFile.AddRange(buffer);
}
fileStream.Close();
}
catch (Exception ex)
{
throw; // you need to change this
}
file.UploadFromByteArray(allFile.ToArray(), 0, byteArrayFile.Count);
// Not sure about byteArrayFile.Count.. it should work
}
According to your description and codes, I suggest you could use Steam.CopyTo to copy the stream to the local memoryStream firstly, then upload the MemoryStream to azure file storage.
More details, you could refer to below codes:
I just change the DownloadFile method to test it.
HttpStatusCode status;
using (var client = new HttpClient())
{
client.Timeout = TimeSpan.FromSeconds(10); // candidate for .config setting
// client.DefaultRequestHeaders.Add("User-Agent", USER_AGENT);
//here I use my blob file to test it
var request = new HttpRequestMessage(HttpMethod.Get, "https://xxxxxxxxxx.blob.core.windows.net/media/secondblobtest-eypt.txt");
var sendTask = client.SendAsync(request, HttpCompletionOption.ResponseHeadersRead);
var response = sendTask.Result; // not ensuring success here, going to handle error codes without exceptions
status = response.StatusCode;
if (status == HttpStatusCode.OK)
{
MemoryStream ms = new MemoryStream();
var httpStream = response.Content.ReadAsStreamAsync().Result;
httpStream.CopyTo(ms);
ms.Position = 0;
WriteFile("aaa", "testaa", ms);
// hash = HashGenerator.GetMD5HashFromStream(httpStream);
}
}
I had a similar problem and got to find out that the UploadFromStream method only works with buffered streams. Nevertheless I was able to successfully upload files to azure storage by using a MemoryStream. I don't think this to be a very good solution as you are using up your memory resources by copying the content of the file stream to memory before handing it to the azure stream. What I have come up with is a way of writing directly to an azure stream by using instead the OpenWriteAsync method to create the stream and then a simple CopyToAsync from the source stream.
CloudStorageAccount storageAccount = CloudStorageAccount.Parse( "YourAzureStorageConnectionString" );
CloudFileClient fileClient = storageAccount.CreateCloudFileClient();
CloudFileShare share = fileClient.GetShareReference( "YourShareName" );
CloudFileDirectory root = share.GetRootDirectoryReference();
CloudFile file = root.GetFileReference( "TheFileName" );
using (CloudFileStream fileWriteStream = await file.OpenWriteAsync( fileMetadata.FileSize, new AccessCondition(),
new FileRequestOptions { StoreFileContentMD5 = true },
new OperationContext() ))
{
await fileContent.CopyToAsync( fileWriteStream, 128 * 1024 );
}
I have an FTP server from which I pull and parse data and download images. I've never had an issue pulling text files (my data) but there are 2 directories that I pull images from and one works without a hitch using code identical to below (though obviously with a different Uri and image name array). In order to get a list of the images I have a function which pulls another file and returns an array of the image names and then I put this array of image names sequentially on the end of a base URI in a loop to download all images.
The code at the bottom is my function for pulling the text file with the image names. It has it's own URI and comes from a different part of the directory structure. The interesting thing is that if I manually create an array of images in a known location and feed my image pull code, it works fine but if I run this array building FTP function my image pulls timeout in the GetResponse() line. I've shortened the timeout so I don't have to wait so long for it to fail but it's not too short since as I said it works fine with a hard coded list of image URIs. I've searched here and tried several things thinking I wasn't releasing resources correctly but I cannot figure out what's happening.
Sorry I'm not an expert at this but I have lots of similar code in this app and this is the first time I've had trouble with FTPing.
One note, I have an earlier version of this app which works fine but I must admit I have been very lax in my source version control and don't have the old code (doh!). I changed something that is now killing it.
//Open a web request based on the URI just built from user selections
FtpWebRequest imageRequest = (FtpWebRequest)WebRequest.Create(imageUri);
imageRequest.UsePassive = false;
imageRequest.UseBinary = true;
imageRequest.KeepAlive = false;
imageRequest.Timeout = 2000;
//Act on that webrequest based on what the user wants to do (i.e. list a directory, list file, download, etc)
imageRequest.Method = WebRequestMethods.Ftp.DownloadFile;
try
{
//Now get the response from the tool as a stream.
FtpWebResponse imageResponse = (FtpWebResponse)imageRequest.GetResponse();
//Thread.Sleep(1000);
Stream imageResponseStream = imageResponse.GetResponseStream();
byte[] buffer = new byte[2048];
FileStream fs = new FileStream(newPath + #"\" + templateImageArray2[x].Split(' ')[0], FileMode.Create);
int ReadCount = imageResponseStream.Read(buffer, 0, buffer.Length);
while (ReadCount > 0)
{
fs.Write(buffer, 0, ReadCount);
ReadCount = imageResponseStream.Read(buffer, 0, buffer.Length);
}
fs.Close();
imageResponseStream.Close();
}
catch (WebException r)
{
if (r.Status == WebExceptionStatus.Timeout)
{
//Obtain more detail on error:
var response = (FtpWebResponse)r.Response;
FtpStatusCode errorCode = response.StatusCode;
string errorMessage = response.StatusDescription;
MessageBox.Show(errorMessage);
//goto RETRY;
}
MessageBox.Show(r.Message);
break;
}
string[] tempIDPText = new string[0];
Uri fileUriLocal = new Uri("ftp://srvrname:password#" + tempIpAddress + "/%2fROOT/DEVICE/HD/" + comboClass.Text + "/data/" + dirName1 + "/" + dirName2 + ".img");
//Open a web request based on the URI just built from user selections
FtpWebRequest requestLocal = (FtpWebRequest)WebRequest.Create(fileUriLocal);
requestLocal.UsePassive = false;
requestLocal.UseBinary = true;
requestLocal.KeepAlive = false;
//Act on that webrequest based on what the user wants to do (i.e. list a directory, list file, download, etc)
requestLocal.Method = WebRequestMethods.Ftp.DownloadFile;
try
{
//Now get the response from the tool as a stream.
FtpWebResponse responseLocal = (FtpWebResponse)requestLocal.GetResponse();
Stream responseStream = responseLocal.GetResponseStream();
StreamReader reader = new StreamReader(responseStream);
//Now read an individual file and fill the array with the chamber status
int y = 0;
while (!reader.EndOfStream)
{
if (reader.EndOfStream)
break;
else
{
Array.Resize(ref tempIDPText, tempIDPText.Length + 1);
tempIDPText[tempIDPText.Length - 1] = reader.ReadLine();
}
}
reader.Close();
responseStream.Close();
responseLocal.Close();
ServicePoint srvrPoint = ServicePointManager.FindServicePoint(fileUriLocal);
MethodInfo ReleaseConns = srvrPoint.GetType().GetMethod
("ReleaseAllConnectionGroups",
BindingFlags.Instance | BindingFlags.NonPublic);
ReleaseConns.Invoke(srvrPoint, null);
}
catch (WebException r)
{
//Obtain more detail on the error;
var response = (FtpWebResponse)r.Response;
FtpStatusCode errorCode = response.StatusCode;
string errorMessage = response.StatusDescription;
MessageBox.Show(errorCode.ToString());
MessageBox.Show("When getting file Dates: " + errorMessage.ToString());
}
return tempIDPText;