How to upload file to Azure DataLake through API? - c#

Net core application. I am trying to upload file to data lake through API. I have below controller method which accepts file.
[HttpPost]
public async Task<IActionResult> Upload(IFormFile files)
{
var path = Path.Combine(Directory.GetCurrentDirectory(), "Uploads", files.FileName);
var stream = new FileStream(path, FileMode.Create);
string containerName = "raw";
DataLakeServiceClient dataLakeServiceClient = _dataLakeRepository.GetDataLakeServiceClient("test");
DataLakeFileSystemClient dataLakeFileSystemClient = _dataLakeRepository.GetFileSystem(dataLakeServiceClient, containerName);
await _dataLakeRepository.UploadFile(dataLakeFileSystemClient, "directory2", "text1.txt", stream);
return Ok();
}
I have below DataLake method which will upload file to data lake.
public async Task UploadFile(DataLakeFileSystemClient fileSystemClient, string directoryName, string fileName, Stream content)
{
DataLakeDirectoryClient directoryClient = fileSystemClient.GetDirectoryClient(directoryName);
DataLakeFileClient fileClient = await directoryClient.CreateFileAsync(fileName);
long fileSize = content.Length;
await fileClient.AppendAsync(content, offset: 0);
await fileClient.FlushAsync(position: fileSize);
}
Below method to get file system client
public DataLakeFileSystemClient GetFileSystem(DataLakeServiceClient serviceClient, string FileSystemName)
{
return serviceClient.GetFileSystemClient(FileSystemName);
}
I tried to upload file and In below line
await fileClient.AppendAsync(content, offset: 0);
I got below error
Azure.RequestFailedException: The value for one of the HTTP headers is not in the correct format.
Status: 400 (The value for one of the HTTP headers is not in the correct format.)
ErrorCode: InvalidHeaderValue
Also when I debug I see content.Length is also zero. I think I am missing something in stream because I am having some issue with stream. I am not able to figure out the issue. Can someone help me to fix this. Any help would be appreciated. Thanks

After read official doc, we can find the sample code use FileStream.
So you should convert Stream to FileStream .

Related

Return File not forcing download of file in .NET core API

I have a .NET Core API trying to return a file for the browser to download when triggered.
[HttpGet("DownloadFile/{fileId}")]
public async Task<ActionResult> DownloadFile(string fileId, CancellationToken cancellationToken = default)
{
var file = await _fileUploadService.DownloadFile(Guid.Parse(fileId), _ambientState.UserId);
[![enter image description here][1]][1]
var path = Path.Combine(
Directory.GetCurrentDirectory(),
"fileuploads", file.FilePath);
byte[] fileBytes = System.IO.File.ReadAllBytes(path);
return File(fileBytes, System.Net.Mime.MediaTypeNames.Application.Octet, "event-doc" + Path.GetExtension(path));
}
But this doesn't force the download of the file I have tried many things but nothing causes the file to be downloaded by the browser. There are no errors either. The call to the API succeeds but no file is downloaded
You'll need to add a content-disposition header before you return the action result.
var header = new ContentDispositionHeaderValue("attachment")
{
FileName = document.FileName
};
Response.Headers.Add(HeaderNames.ContentDisposition, header.ToString());

IFormFileCollection is null when adding file to Flurl multi-part POST request

I am trying to upload a file using Flurl using AddFile.
The resulting IFormFileCollection is null, although I am able to see the item when looking at Request.Form.Files[0] with the right content length.
Creating the request:
public Task<HttpResponseMessage> UploadImage(string fileName, MemoryStream stream)
{
stream.Position = 0;
_baseUrl
.AppendPathSegments("uploadImage")
.PostMultipartAsync(mp => mp
.AddFile("files", stream, fileName))
}
Handling the request:
[HttpPost]
[Route("uploadImage")]
public async Task<HttpResponseMessage> UploadImages([FromForm] IFormFileCollection files)
{
//files is null, but Request.Form.Files[0] in the immediate window shows the file.
}
A common problem seems to be a mismatch in the name of the parameter and the name in the Content-Disposition header, but I updated them to both be files and I am still having the same issue.
That is strange that it works on my side :
MemoryStream ms = new MemoryStream();
using (FileStream file = new FileStream("txt.txt", FileMode.Open, FileAccess.Read))
file.CopyTo(ms);
ms.Position = 0;
var _baseUrl = "https://localhost:44392/";
var result = await _baseUrl
.AppendPathSegments("uploadImage")
.PostMultipartAsync(mp => mp
.AddFile("files", ms, "txt.txt"));
Result :
Please firstly try with a clean file and use await for handing request .

Upload a pdf file to azure blob using azure function

I am trying to upload a PDF file from postman and trigger the azure function to upload the PDF file into azure blob storage. But when i try to open the PDF file it is always empty.
I tried to convert the file into memory stream and upload it into azure blob. The file gets uploaded but when i try to open the file it will be blank.
public static async Task<HttpResponseMessage> Run([HttpTrigger(AuthorizationLevel.Function, "get", "post", Route = null)]HttpRequestMessage req, TraceWriter log)
{
log.Info(req.Content.ToString());
string Message = "";
log.Info("Test storage conn string" + req.Content.Headers.ContentDisposition.ToString());
string contentType = req.Content.Headers?.ContentType?.MediaType;
log.Info("contentType : " + req.Content.IsMimeMultipartContent());
string name = Guid.NewGuid().ToString("n");
log.Info("Name" + name);
string body;
body = await req.Content.ReadAsStringAsync();
log.Info("body" + body.Substring(body.IndexOf("filename=\""),body.IndexOf("pdf")- body.IndexOf("filename=\"")));
//Upload a file to Azure blob
string storageConnectionString = "xxxx";
//DirectoryInfo directoryInfo = new DirectoryInfo("D:\\Upload_Files");
// var files = directoryInfo.EnumerateFiles();
// Retrieve storage account from connection string.
CloudStorageAccount storageAccount = CloudStorageAccount.Parse(storageConnectionString);
// Create the blob client.
CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();
// Retrieve reference to a previously created container.
CloudBlobContainer container = blobClient.GetContainerReference("docstorage");
//foreach (FileInfo inputFile in files)
//{
CloudBlockBlob blockBlob = container.GetBlockBlobReference("Test\\" + name+".pdf");//write name here
//blockBlob.Properties.ContentType = "application/pdf";
//blockBlob.UploadFromFile(inputFile.FullName);
using (Stream stream = new MemoryStream(Encoding.UTF8.GetBytes(body)))
{
log.Info("streaming : ");
await blockBlob.UploadFromStreamAsync(stream);
}
//}
return Message == null
? req.CreateResponse(HttpStatusCode.BadRequest, "Error")
: req.CreateResponse(HttpStatusCode.OK, "Doc Uploaded Successfully");
}
I want to open the PDF file as it is from the blob. I see that i am able to upload text file and when i download i can see the content but when i upload pdf file i dont see the content
Calling .ReadAsStringAsync on a binary document wont work - you have to call ReadAsByteArrayAsync or ReadAsStreamAsync.
var body = await req.Content.ReadAsByteArrayAsync();
...
using (Stream stream = new MemoryStream(body))
{
await blockBlob.UploadFromStreamAsync(stream);
}
OR
var body2 = await req.Content.ReadAsStreamAsync();
body.Position = 0;
...
await blockBlob.UploadFromStreamAsync(body);
It's really simple to do something like that. Everything relative to bindings should be declared in the function parameters so, having this in mind, you have to declare your blob stream as a parameter. Check this as an example:
public static async Task<string> Run(
[HttpTrigger(AuthorizationLevel.Function, "post", Route = null)] HttpRequest req,
[Blob("azurefunctions/test.pdf", FileAccess.Write)] Stream blob,
ILogger log)
Please, note the second parameter called blob is declared as a Stream to be able to save the content read from the input. The second point is the attribute decorating the parameter, Blob allows to define several aspects of the new blob file that will be uploaded in our Azure Storage service. As you can see, the container is called azurefunctions and the file will be called test.pdf.
In order to save the content you can use the following code:
byte[] content = new byte[req.Body.Length];
await req.Body.ReadAsync(content, 0, (int)req.Body.Length);
await blob.WriteAsync(content, 0, content.Length);
Hope this can be helpful for your question.
These are useful links to check and test your code:
Azure Blob storage bindings for Azure Functions
How to send multipart/form-data with PowerShell Invoke-RestMethod

C# MVC Download Big File from S3 Async

I have to download a file from aws S3 async. I have a anchor tag, on clicking it a method will be hit in a controller for download. The file should be start downloading at the bottom of the browser, like other file download.
In View
Click here
In Controller
public void action()
{
try
{
AmazonS3Client client = new AmazonS3Client(accessKeyID, secretAccessKey);
GetObjectRequest req = new GetObjectRequest();
req.Key = originalName;
req.BucketName = ConfigurationManager.AppSettings["bucketName"].ToString() + DownloadPath;
FileInfo fi = new FileInfo(originalName);
string ext = fi.Extension.ToLower();
string mimeType = ReturnmimeType(ext);
var res = client.GetObject(req);
Stream responseStream = res.ResponseStream;
Stream response = responseStream;
return File(response, mimeType, downLoadName);
}
catch (Exception)
{
failure = "File download failed. Please try after some time.";
}
}
The above function makes the browser to load until the file is fully downloaded. Then only the file is visible at the bottom. I cant see the how mb is downloading.
Thanks in advance.
You must send ContentLength to client in order to display a progress. Browser has no information about how much data it will receive.
If you look at source of FileStreamResult class, used by File method, it does not inform client about "Content-Length". https://aspnetwebstack.codeplex.com/SourceControl/latest#src/System.Web.Mvc/FileStreamResult.cs
Replace this,
return File(response, mimeType, downLoadName);
with
return new FileStreamResultEx(response, res.ContentLength, mimeType, downloadName);
public class FileStreamResultEx : ActionResult{
public FileStreamResultEx(
Stream stream,
long contentLength,
string mimeType,
string fileName){
this.stream = stream;
this.mimeType = mimeType;
this.fileName = fileName;
this.contentLength = contentLength;
}
public override void ExecuteResult(
ControllerContext context)
{
var response = context.HttpContext.Response;
response.BufferOutput = false;
response.Headers.Add("Content-Type", mimeType);
response.Headers.Add("Content-Length", contentLength.ToString());
response.Headers.Add("Content-Disposition","attachment; filename=" + fileName);
using(stream) {
stream.CopyTo(response.OutputStream);
}
}
}
Alternative
Generally this is a bad practice to download and deliver S3 file from your server. You will be charged twice bandwidth on your hosting account. Instead, you can use signed URLs to deliver non public S3 objects, with few seconds of time to live. You could simply use Pre-Signed-URL
public ActionResult Action(){
try{
using(AmazonS3Client client =
new AmazonS3Client(accessKeyID, secretAccessKey)){
var bucketName =
ConfigurationManager.AppSettings["bucketName"]
.ToString() + DownloadPath;
GetPreSignedUrlRequest request1 =
new GetPreSignedUrlRequest(){
BucketName = bucketName,
Key = originalName,
Expires = DateTime.Now.AddMinutes(5)
};
string url = client.GetPreSignedURL(request1);
return Redirect(url);
}
}
catch (Exception)
{
failure = "File download failed. Please try after some time.";
}
}
As long as object do not have public read policy, objects are not accessible to users without signing.
Also, you must use using around AmazonS3Client in order to quickly dispose networks resources, or just use one static instance of AmazonS3Client that will reduce unnecessary allocation and deallocation.
As i understand, you want to make something like "reverse proxy" from your server to S3. Very userful article how to do that with Nginx you can find here: https://stackoverflow.com/a/44749584

Google drive api v3 response null

I want to try upload file google drive and my code is here:
public static File UploadFile(DriveService service, string fileName, string filePath, string description, string parent)
{
var fileMetadata = new File
{
Name = fileName,
MimeType = GetMimeType(fileName),
Description = description,
OriginalFilename = fileName,
};
FilesResource.CreateMediaUpload request;
using (var stream = new System.IO.FileStream(filePath, System.IO.FileMode.Open))
{
request = service.Files.Create(fileMetadata, stream, GetMimeType(fileName));
request.Fields = "id";
request.Alt=FilesResource.CreateMediaUpload.AltEnum.Json;
request.Upload();
}
var file = request.ResponseBody;
Console.WriteLine("File ID: " + file.Id);
return null;
}
But Response body is always null and i cant upload. By the way i saw error in this picture:
google-api-dotnet-client 1.16.0.0 gzip the format of the invalid value
Anyone idea how to solve this problem and upload files google drive.
It appears V3 has replaced InsertMediaUpload for CreateMediaUpload and when using a service account at least the ResponseBody will always be returned as NULL.
This makes it difficult to determine the ID of the file and later find it to modify permissions or do anything else.

Categories