We Keep Coding

How to upload file to Azure DataLake through API?

Net core application. I am trying to upload file to data lake through API. I have below controller method which accepts file.
[HttpPost]
public async Task<IActionResult> Upload(IFormFile files)
{
var path = Path.Combine(Directory.GetCurrentDirectory(), "Uploads", files.FileName);
var stream = new FileStream(path, FileMode.Create);
string containerName = "raw";
DataLakeServiceClient dataLakeServiceClient = _dataLakeRepository.GetDataLakeServiceClient("test");
DataLakeFileSystemClient dataLakeFileSystemClient = _dataLakeRepository.GetFileSystem(dataLakeServiceClient, containerName);
await _dataLakeRepository.UploadFile(dataLakeFileSystemClient, "directory2", "text1.txt", stream);
return Ok();
}
I have below DataLake method which will upload file to data lake.
public async Task UploadFile(DataLakeFileSystemClient fileSystemClient, string directoryName, string fileName, Stream content)
{
DataLakeDirectoryClient directoryClient = fileSystemClient.GetDirectoryClient(directoryName);
DataLakeFileClient fileClient = await directoryClient.CreateFileAsync(fileName);
long fileSize = content.Length;
await fileClient.AppendAsync(content, offset: 0);
await fileClient.FlushAsync(position: fileSize);
}
Below method to get file system client
public DataLakeFileSystemClient GetFileSystem(DataLakeServiceClient serviceClient, string FileSystemName)
{
return serviceClient.GetFileSystemClient(FileSystemName);
}
I tried to upload file and In below line
await fileClient.AppendAsync(content, offset: 0);
I got below error
Azure.RequestFailedException: The value for one of the HTTP headers is not in the correct format.
Status: 400 (The value for one of the HTTP headers is not in the correct format.)
ErrorCode: InvalidHeaderValue
Also when I debug I see content.Length is also zero. I think I am missing something in stream because I am having some issue with stream. I am not able to figure out the issue. Can someone help me to fix this. Any help would be appreciated. Thanks

After read official doc, we can find the sample code use FileStream.
So you should convert Stream to FileStream .

How to save the response in the Postman same as the name I mentioned it?

This piece of code will put all the data(string) as an XML format in the Postman. But when I try to save the response in the Postman so, it will save as response.xml, which I don't want. I want that Postman will save my XML content in the form of filename.xml.
So, how should I proceed with? I have to google it and found that something like attachment, content-type, and all must be used. But, didn't solve my case. Please help with your suggestion.
For simplicity, I have reduced the code. Here, graphics is string datatype.
public ActionResult GetXML()
{
string filename = "Demo";
return Content(graphics, "application/xml");
}

You need to set Content-Disposition header so the that browser prompts for saving response.
You can do that by using following approach.
public IActionResult Getxml()
{
var xmlData =
"<records><record><Name>Camacho, Sydnee Q.</Name><Id>1</Id><Age>19</Age><City>Podolsk</City></record><record><Name>Bowman, Lester V.</Name><Id>2</Id><Age>21</Age><City>Padang</City></record></records>";
//sampleFile.xml can be replaced by any filename of your choice.
var fileName = "sampleFile.xml";
Response.Headers.Add("Content-Disposition", $"attachment; filename={fileName}");
return Content(xmlData, "application/xml", Encoding.UTF8);
}
With this change when this URL is browsed in the browser it will prompt save dialog with sampleFile.xml populated as filename.
Also when you save the response from postman it will show sampleFile.xml populated as file name in the dialog.
I hope this will help you solve your issue.

I would suggest to use the FileContentResult for a file download.
public ActionResult GetXML()
{
var fileName = "Demo.xml";
var xml = "...";
return new FileContentResult(Encoding.UTF8.GetBytes(xml), "application/xml; charset=utf-8")
{
FileDownloadName = fileName,
};
}

You need to set the response headers so that the download client automatically treats the content as a "file", the key is the ContentDisposition header, this is how we pass the filename back.
public HttpResponseMessage GetXMLFile()
{
string filename = "Demo";
var byteArray = System.Text.Encoding.UTF8.GetBytes(graphics);
HttpResponseMessage result = new HttpResponseMessage(HttpStatusCode.OK)
{
Content = new StreamContent(new MemoryStream(byteArray), byteArray.Length)
};
result.Content.Headers.ContentType = new MediaTypeHeaderValue("application/xml");
result.Content.Headers.ContentDisposition = new ContentDispositionHeaderValue("attachment") //attachment will force download
{
FileName = filename
};
return result;
}

Return File not forcing download of file in .NET core API

I have a .NET Core API trying to return a file for the browser to download when triggered.
[HttpGet("DownloadFile/{fileId}")]
public async Task<ActionResult> DownloadFile(string fileId, CancellationToken cancellationToken = default)
{
var file = await _fileUploadService.DownloadFile(Guid.Parse(fileId), _ambientState.UserId);
[![enter image description here][1]][1]
var path = Path.Combine(
Directory.GetCurrentDirectory(),
"fileuploads", file.FilePath);
byte[] fileBytes = System.IO.File.ReadAllBytes(path);
return File(fileBytes, System.Net.Mime.MediaTypeNames.Application.Octet, "event-doc" + Path.GetExtension(path));
}
But this doesn't force the download of the file I have tried many things but nothing causes the file to be downloaded by the browser. There are no errors either. The call to the API succeeds but no file is downloaded

You'll need to add a content-disposition header before you return the action result.
var header = new ContentDispositionHeaderValue("attachment")
{
FileName = document.FileName
};
Response.Headers.Add(HeaderNames.ContentDisposition, header.ToString());

Twilio download recording to file throwing error on JsonConvert.DeserializeObject()

I'm trying to download my recordings on Twilio to a file on my servers local file system (so I can send them to another storage location), but following the code that I've found is throwing an error on the JsonConvert.DeserializeObject() call.
The code that I found is here (called "Retrieve the actual recording media"): https://www.twilio.com/docs/video/api/recordings-resource#filter-by-participant-sid
Here's the code:
static void Main(string[] args)
{
// Find your Account SID and Auth Token at twilio.com/console
const string apiKeySid = "SKXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX";
const string apiKeySecret = "your_api_key_secret";
const string recordingSid = "RTXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX";
const string uri = $"https://video.twilio.com/v1/Recordings/{recordingSid}/Media";
var request = (HttpWebRequest)WebRequest.Create(uri);
request.Headers.Add("Authorization", "Basic " + Convert.ToBase64String(Encoding.ASCII.GetBytes(apiKeySid + ":" + apiKeySecret)));
request.AllowAutoRedirect = false;
string responseBody = new StreamReader(request.GetResponse().GetResponseStream()).ReadToEnd();
var mediaLocation = JsonConvert.DeserializeObject<Dictionary<string, string>>(responseBody)["redirect_to"];
Console.WriteLine(mediaLocation);
new WebClient().DownloadFile(mediaLocation, $"{recordingSid}.out");
}
And here's my version:
var twilioRecordingUri = $"https://api.twilio.com/2010-04-01/Accounts/{recording.AccountSid}/Recordings/{recording.Sid}.mp3?Download=true";
var request = (HttpWebRequest)WebRequest.Create(new Uri(twilioRecordingUri));
request.Headers.Add("Authorization", "Basic " + Convert.ToBase64String(Encoding.ASCII.GetBytes($"{apiKeySid}:{apiKeySecret}")));
request.ContentType = "audio/mpeg";
//request.Accept = "audio/mpeg";
request.AllowAutoRedirect = false;
var responseBody = new StreamReader(request.GetResponse().GetResponseStream()).ReadToEnd();
var deserialized = JsonConvert.DeserializeObject<Dictionary<string, string>>(responseBody);
var mediaLocation = deserialized["redirect_to"];
new WebClient().DownloadFile(mediaLocation, $"{recording.Sid}.out");
But executing that code, it fails on the JsonConvert.Deserialize(), like I mentioned; it throws this generic Json error:
Unexpected character encountered while parsing value: �. Path '', line
0, position 0.
Hovering over my "responseBody" variable does show that it's a really long string of funky characters.
My thought was that I should be adding either the "Accept" or "Content-type" to "audio/mpeg" since that's the type of file that I'm trying to retrieve. But when checking Dev Tools at both the request and response headers, neither the Accept or Content-type ever get my audio/mpeg setting that I just specified.
What's wrong with this code here?
Edit: for anyone that noticed the download URL is different from Twilio's example, I found this page that had the updated URL: How do I get a call recording url in twilio when programming in PHP?

I'm only posting this answer to show what the "working" version looks like. It was #Grungondola 's answer that prompted me. Thanks goes to him (as well as the accepted answer).
private async Task DownloadRecording(RecordingResource recording, string fileName)
{
if (string.IsNullOrEmpty(fileName))
throw new ArgumentNullException("fileName is required when downloading a recording.");
var twilioRecordingUri = $"https://api.twilio.com/2010-04-01/Accounts/{recording.AccountSid}/Recordings/{recording.Sid}.mp3?Download=false";
var request = (HttpWebRequest)WebRequest.Create(new Uri(twilioRecordingUri));
request.Headers.Add("Authorization", "Basic " + Convert.ToBase64String(Encoding.ASCII.GetBytes($"{apiKeySid}:{apiKeySecret}")));
request.ContentType = "audio/mpeg";
request.AllowAutoRedirect = false;
var stream = request.GetResponse().GetResponseStream();
var virtualPath = HttpContext.Server.MapPath(fileName);
var fileStream = new FileStream(virtualPath, FileMode.Create);
await stream.CopyToAsync(fileStream);
fileStream.Close();
}

It looks like the call you're trying to make is to download an .mp3 file and not a Dictionary<string, string>, so it's likely hitting an error when attempting to deserialize a string into a type that it doesn't match. What you're probably seeing as a result is a Base64 string, especially based on your description. Without seeing at least a sample of the data, I can't know for sure, but I'd guess that you're downloading the raw .mp3 file instead of the file information with location (redirect_to).
If the result is a pure Base64 string, you should be able to convert it to an array of bytes and write that directly to a file with whatever filename you want. That should get you the mp3 file that you want.

C# MVC Download Big File from S3 Async

I have to download a file from aws S3 async. I have a anchor tag, on clicking it a method will be hit in a controller for download. The file should be start downloading at the bottom of the browser, like other file download.
In View
Click here
In Controller
public void action()
{
try
{
AmazonS3Client client = new AmazonS3Client(accessKeyID, secretAccessKey);
GetObjectRequest req = new GetObjectRequest();
req.Key = originalName;
req.BucketName = ConfigurationManager.AppSettings["bucketName"].ToString() + DownloadPath;
FileInfo fi = new FileInfo(originalName);
string ext = fi.Extension.ToLower();
string mimeType = ReturnmimeType(ext);
var res = client.GetObject(req);
Stream responseStream = res.ResponseStream;
Stream response = responseStream;
return File(response, mimeType, downLoadName);
}
catch (Exception)
{
failure = "File download failed. Please try after some time.";
}
}
The above function makes the browser to load until the file is fully downloaded. Then only the file is visible at the bottom. I cant see the how mb is downloading.
Thanks in advance.

You must send ContentLength to client in order to display a progress. Browser has no information about how much data it will receive.
If you look at source of FileStreamResult class, used by File method, it does not inform client about "Content-Length". https://aspnetwebstack.codeplex.com/SourceControl/latest#src/System.Web.Mvc/FileStreamResult.cs
Replace this,
return File(response, mimeType, downLoadName);
with
return new FileStreamResultEx(response, res.ContentLength, mimeType, downloadName);
public class FileStreamResultEx : ActionResult{
public FileStreamResultEx(
Stream stream,
long contentLength,
string mimeType,
string fileName){
this.stream = stream;
this.mimeType = mimeType;
this.fileName = fileName;
this.contentLength = contentLength;
}
public override void ExecuteResult(
ControllerContext context)
{
var response = context.HttpContext.Response;
response.BufferOutput = false;
response.Headers.Add("Content-Type", mimeType);
response.Headers.Add("Content-Length", contentLength.ToString());
response.Headers.Add("Content-Disposition","attachment; filename=" + fileName);
using(stream) {
stream.CopyTo(response.OutputStream);
}
}
}
Alternative
Generally this is a bad practice to download and deliver S3 file from your server. You will be charged twice bandwidth on your hosting account. Instead, you can use signed URLs to deliver non public S3 objects, with few seconds of time to live. You could simply use Pre-Signed-URL
public ActionResult Action(){
try{
using(AmazonS3Client client =
new AmazonS3Client(accessKeyID, secretAccessKey)){
var bucketName =
ConfigurationManager.AppSettings["bucketName"]
.ToString() + DownloadPath;
GetPreSignedUrlRequest request1 =
new GetPreSignedUrlRequest(){
BucketName = bucketName,
Key = originalName,
Expires = DateTime.Now.AddMinutes(5)
};
string url = client.GetPreSignedURL(request1);
return Redirect(url);
}
}
catch (Exception)
{
failure = "File download failed. Please try after some time.";
}
}
As long as object do not have public read policy, objects are not accessible to users without signing.
Also, you must use using around AmazonS3Client in order to quickly dispose networks resources, or just use one static instance of AmazonS3Client that will reduce unnecessary allocation and deallocation.

As i understand, you want to make something like "reverse proxy" from your server to S3. Very userful article how to do that with Nginx you can find here: https://stackoverflow.com/a/44749584