We Keep Coding

C# HttpClient save response with MIME "text/plain" as an UTF-8 string

I'm sending a request with HttpClient to a remote endpoint. I want to download the content and save it to a file as an UTF-8 string.
If the server would respond with the proper Content-Type text/plain; charset=utf-8, then the following code processes it just fine:
HttpClient client = new();
HttpResponseMessage res = await client.GetAsync(url);
string text = await res.Content.ReadAsStringAsync();
File.WriteAllText("file.txt", text);
However, the server always returns the basic Content-Type text/plain and I'm unable to get that as an UTF-8 string.
HttpClient cl = new();
HttpResponseMessage res = await cl.GetAsync(url);
string attempt1 = await res.Content.ReadAsStringAsync();
string attempt2 = Encoding.UTF8.GetString(await res.Content.ReadAsByteArrayAsync());
Stream stream = await res.Content.ReadAsStreamAsync();
byte[] bytes = ((MemoryStream)stream).ToArray();
string attempt3 = Encoding.UTF8.GetString(bytes);
I tried all three of these approaches, all resulted in scrambled characters due to the encoding mismatch. I don't have control over the server, so I can't change the headers.
Is there any way to force HttpClient to parse it as UTF-8? Why are the manual approaches not working?
I've built a Cloudflare worker to demonstrate this behavior and allow you to easily debug:
https://headers.briganreiz.workers.dev/charset-in-header
https://headers.briganreiz.workers.dev/no-charset
Edit: Turns out it was the GZip compression on the main server which I didn't notice. This question solved it for me: Decompressing GZip Stream from HTTPClient Response

I find it works well with these different classes WebRequest and HttpWebResponse. I have not added plumbing for resp.StatusCode etc but obviously presuming all went well is a tad naive.
Give it a go i am sure You'll find the WebRequest and HttpWebResponse more capable for dynamic requests (?)
var req = WebRequest.CreateHttp(url)
var getResponse = req.GetResponseAsync();
getResponse.Wait(ResponseTimeoutMilliseconds);
var resp = (HttpWebResponse)getResponse.Result;
using (Stream responseStream = resp.GetResponseStream())
{
var reader = new StreamReader(responseStream, Encoding.UTF8);
string content = reader.ReadToEnd();
}
Obviously once you have things working, you should absolutely use the ..Async versions but for debugging, since we already waited for response it is more convenient to simply step through i find, feel free to not take that middle step :)

Malformed multipart body when uploading file to Google Drive

Having a bit of trouble trying to resolve the issue with file upload to Google Drive using their /upload endpoint. I keep getting Malformed multipart body. error even when I try to upload simple plain text as a file.
The following .net c# code is used to create the request:
string fileName = "test.txt";
string fileContent = "The quick brown fox jumps over the lazy dog";
var fileStream = GenerateStreamFromString(fileContent); // simple text string to Stream conversion
var streamContent = new StreamContent(fileStream);
streamContent.Headers.ContentType = new MediaTypeHeaderValue("text/plain");
var multiPartFormDataContent = new MultipartFormDataContent("not_so_random_boundary");
// rfc2387 headers with boundary
multiPartFormDataContent.Headers.Remove("Content-Type");
multiPartFormDataContent.Headers.TryAddWithoutValidation("Content-Type", "multipart/related; boundary=" + "not_so_random_boundary");
// metadata part
multiPartFormDataContent.Add(new StringContent("{\"name\":\"" + fileName + "\",\"mimeType\":\"text/plain\",\"parents\":[\"" + folder.id + "\"]}", Encoding.UTF8, "application/json"));
// media part (file)
multiPartFormDataContent.Add(streamContent);
var response_UploadFile = await httpClient.PostAsync(string.Format("https://www.googleapis.com/upload/drive/v3/files?uploadType=multipart"), multiPartFormDataContent);
I log the following Request:
Method: POST,
RequestUri: 'https://www.googleapis.com/upload/drive/v3/files?uploadType=multipart',
Version: 1.1,
Content: System.Net.Http.MultipartFormDataContent,
Headers: { Authorization: Bearer /*snip*/ Content-Type: multipart/related; boundary=not_so_random_boundary }
with following request content (pretified):
--not_so_random_boundary
Content-Type: application/json; charset=utf-8
Content-Disposition: form-data
{"name":"test.txt","mimeType":"text/plain","parents":["/*snip*/"]}
--not_so_random_boundary
Content-Type: text/plain
Content-Disposition: form-data
The quick brown fox jumps over the lazy dog
--not_so_random_boundary--
I've spent the entire day on this and it got me to this point. I have a hunch that issue is something silly but I just can't figure it out.
Could someone throw their eyes over this perhaps you can spot where I made a mistake that would be very helpful?
###ref:
Send a multipart upload request
RFC 2387

Thanks to #Tanaike suggestion we found the problem with my code.
Turns out while it is not specifically mentioned in the documentation (or any code examples) but adding Content-Disposition: form-data; name="metadata" to the StringContent part of the request body makes all the difference.
The final request can be rewritten as follows:
// sample file (controlled test example)
string fileName = "test.txt";
string fileType = "text/plain";
string fileContent = "The quick brown fox jumps over the lazy dog";
var fileStream = GenerateStreamFromString(fileContent); // test file
// media part (file)
//var fileStream = File.Open(path_to_file, FileMode.Open, FileAccess.Read); // you should read file from disk
var streamContent = new StreamContent(fileStream);
streamContent.Headers.ContentDisposition = new ContentDispositionHeaderValue("form-data");
streamContent.Headers.ContentDisposition.Name = "\"file\"";
// metadata part
var stringContent = new StringContent("{\"name\":\"" + fileName + "\",\"mimeType\":\"" + fileType + "\",\"parents\":[\"" + folder.id + "\"]}", Encoding.UTF8, "application/json");
stringContent.Headers.ContentDisposition = new ContentDispositionHeaderValue("form-data");
stringContent.Headers.ContentDisposition.Name = "\"metadata\"";
var boundary = DataTime.Now.Ticks.ToString(); // or hard code a string like in my previous code
var multiPartFormDataContent = new MultipartFormDataContent(boundary);
// rfc2387 headers with boundary
multiPartFormDataContent.Headers.Remove("Content-Type");
multiPartFormDataContent.Headers.TryAddWithoutValidation("Content-Type", "multipart/related; boundary=" + boundary);
// request body
multiPartFormDataContent.Add(stringContent); // metadata part - must be first part in request body
multiPartFormDataContent.Add(streamContent); // media part - must follow metadata part
var response_UploadFile = await httpClient.PostAsync(string.Format("https://www.googleapis.com/upload/drive/v3/files?uploadType=multipart"), multiPartFormDataContent);
Note that normally one would add file name and content type as part of the StreamContent but these headers are ignored by Google Drive API. This is done deliberatly because the API expects to recieve a metadata object with relevant properties. (the following headers were removed from above code example but will be retained here for future reference)
streamContent.Headers.ContentDisposition.FileName = "\"" + fileName + "\"";
streamContent.Headers.ContentType = new MediaTypeHeaderValue(fileType);
Note that you only need to specify "parents":["{folder_id}"] property if you want to upload file to a subfolder in Google Drive.
Hope this helps someone else in the future.

I think that the structure of the request body for multipart/related might not be correct. So how about modifying as follows?
Modified request body:
--not_so_random_boundary
Content-Type: application/json; charset=utf-8
Content-Disposition: form-data; name="metadata"
{"name":"test.txt","mimeType":"text/plain","parents":["/*snip*/"]}
--not_so_random_boundary
Content-Type: text/plain
Content-Disposition: form-data; name="file"
The quick brown fox jumps over the lazy dog
--not_so_random_boundary--
Please be careful the line breaks for the request body.
Please add name for each part of Content-Disposition.
Note:
Now I could confirm that when above modified request body is used for the endpoint of https://www.googleapis.com/upload/drive/v3/files?uploadType=multipart as POST method, a text file of test.txt which has the content of The quick brown fox jumps over the lazy dog is created.
References:
Content-Type
Content-Disposition
If this didn't work, I apologize.

Another option would be to use the Google .net client library and let it handel the upload for you.
// Upload file Metadata
var fileMetadata = new Google.Apis.Drive.v3.Data.File()
{
Name = "Test hello uploaded.txt",
Parents = new List() {"10krlloIS2i_2u_ewkdv3_1NqcpmWSL1w"}
};
string uploadedFileId;
// Create a new file on Google Drive
await using (var fsSource = new FileStream(UploadFileName, FileMode.Open, FileAccess.Read))
{
// Create a new file, with metadata and stream.
var request = service.Files.Create(fileMetadata, fsSource, "text/plain");
request.Fields = "*";
var results = await request.UploadAsync(CancellationToken.None);
if (results.Status == UploadStatus.Failed)
{
Console.WriteLine($"Error uploading file: {results.Exception.Message}");
}
// the file id of the new file we created
uploadedFileId = request.ResponseBody?.Id;
}
Upload files to google drive

Prevent IDM from downloading automatically in web api

I have a web api method that returns an HttpResponseMessage containing a PDF file. The method looks something like this:
HttpResponseMessage response = Request.CreateResponse(HttpStatusCode.OK);
response.Content = new StreamContent(new FileStream(path, FileMode.Open, FileAccess.Read));
response.Content.Headers.ContentDisposition = new ContentDispositionHeaderValue("attachment");
response.Content.Headers.ContentDisposition.FileName = fileName;
response.Content.Headers.ContentType = new MediaTypeHeaderValue("application/pdf");
return response;
When I call this api from client (which is written in angularJS), the Internet Download Manager automatically catches the PDF file and wants to download it. And because I have a security plan for my project, the IDM automatically requests username and password.
Does anyone have an idea about how I'm supposed to programmatically stop IDM from catching the PDF file?
Update: Here's my angularJS code:
$http.post(url, { transactionId: txId }
, {responseType: 'arraybuffer'})
.success(function (response) {
var reader = new FileReader();
var file = new Blob([response.data], {type: 'application/pdf'});
reader.onload = function (e) {
var printElem = angular.element('#printPdfLink');
printElem.attr('target', '_blank');
printElem.attr('href', reader.result);
printElem.attr('ng-click', '');
};
reader.readAsDataURL(file);
})
.error(function (error) {});

Change the mime type to application/octet-stream as a way to work around your problem. Make sure that the file name includes a proper file extension so that it can be recognized by the client system once downloaded.
Another issue is the attachment disposition of the content which typically forces it to save it as a file download. Change it to inline so that the client can consume it without IDM trying to download it as an attachment.
FileStream stream = new FileStream(path, FileMode.Open, FileAccess.Read);
StreamContent content new StreamContent(stream);
content.Headers.ContentDisposition = new ContentDispositionHeaderValue("inline");
content.Headers.ContentDisposition.FileName = fileName;
content.Headers.ContentType = MediaTypeHeaderValue.Parse("application/octet-stream");
HttpResponseMessage response = Request.CreateResponse(HttpStatusCode.OK);
response.Content = content;
return response;

I have try to use HttpResponseMessage.
If I use ContentDisposition is inline then response break the file. If use attachment then IDM can detect it.
At the end of the day, I found Accept-Ranges header can make download without IDM but it not valid in HttpResponseMessage.
You can try out my code below to make download file without IDM:
[HttpGet]
[Route("~/download/{filename}")]
public void Download(string filename)
{
// TODO lookup file path by {filename}
// If you want to have "." in {filename} you need enable in webconfig
string filePath = "<path>"; // your file path here
byte[] fileBytes = File.ReadAllBytes(filePath);
HttpContext.Current.Response.Clear();
HttpContext.Current.Response.AddHeader("Accept-Ranges", "bytes");
HttpContext.Current.Response.ContentType = "application/octet-stream";
HttpContext.Current.Response.AddHeader("ContentDisposition", "attachment, filename=" + filename);
HttpContext.Current.Response.BinaryWrite(fileBytes);
HttpContext.Current.Response.End();
}
Note: filename parameter serve for download file name so you can config in webconfig if you want to have file extension (disabled by default).

Upload Reference File to Content ID

Has anyone attempted to do this? From what I understand Google doesn't provide a .NET client library for this (the only examples I found were in Python and PHP). The difficulty I'm having is that it seems to want a file upload and a JSON request body in the same request. Any info would be appreciated.

I was able to figure it out. In .NET (using HttpClient), you can use
using (MultipartFormDataContent content = new MultipartFormDataContent("----------" + DateTime.Now.ToString(CultureInfo.InvariantCulture)))
{
content.Add(new StringContent(JsonConvert.SerializeObject(uploadReferenceRequest), Encoding.UTF8, "application/json"));
StreamContent audioContent = new StreamContent(new MemoryStream(buffer));
audioContent.Headers.ContentType = new MediaTypeHeaderValue("application/octet-stream");
content.Add(audioContent);
using (HttpResponseMessage response = await this.Client.PostAsync(url, content))
{
string responseContent = await response.Content.ReadAsStringAsync();
}
}
The key is to set the content type on each section of data, on the first is the "application/json" and the actual sound data is "application/octet-stream". Hope someone else finds this useful.

C# File Download is Corrupt

I've got some C# in a utility for a Web API project. The upload portion of the code works fine; I've verified the file that gets to the server matches the file that was uploaded. However, something is happening in the download that causes the client to view the file as corrupted, and when I do a diff I can see that something went wrong.
Unfortunately, I can't figure out what I'm doing wrong. The relevant parts of the utility are as follows:
public static HttpResponseMessage StreamResponse(this HttpRequestMessage request, Stream stream)
{
if (stream.CanSeek) stream.Position = 0;// Reset stream if possible
HttpResponseMessage response = request.CreateResponse(HttpStatusCode.OK);
response.Content = new StreamContent(stream);
if (stream is FileStream)
{// If this is a FileStream, might as well figure out the content type
string mimeType = MimeMapping.GetMimeMapping(((FileStream)stream).Name);
response.Content.Headers.ContentType = MediaTypeHeaderValue.Parse(mimeType);
}
return response;
}
public static HttpResponseMessage DownloadAs(this HttpResponseMessage response, string fileName)
{
response.Content.Headers.ContentDisposition = new ContentDispositionHeaderValue("attachment");
response.Content.Headers.ContentDisposition.FileName = fileName;
response.Content.Headers.ContentType = MediaTypeHeaderValue.Parse(MimeMapping.GetMimeMapping(fileName));
return response;// For chaining or whatnot
}
My usage in the API controllers is return ResponseMessage(Request.StreamResponse(stream).DownloadAs("Filename.ext"));
I've double checked code for downloading, and this seems to match up with what I've found. What am I doing wrong or what am I missing? It looks like something's wrong with the encoding or charset, but I can't tell what the solution is.

Finally figured out the issue thanks to this Q&A. I was missing the responseType option/parameter in my $http call in the client-side code.