We Keep Coding

.NET HttpClient Image Upload to PlantNet API

I am trying to make a Request to the PlantNet API via .NET HttpClient. I have a FileStream and I am using the StreamContent and when I look via debugger at the content before it is sent it's looking good. However PlantNet response is Unsupported file type for image[0] (jpeg or png).
I tried everything that came in my mind, the same request from VS Code Rest Client is working (with the same file), does anyone have any ideas if the StreamContent is messing somehow with the file data?
HttpResponseMessage responseMessage;
using (MultipartFormDataContent content = new("abcdef1234567890")) //Fixed boundary for debugging
{
content.Add(new StringContent("flower"), "organs");
using Stream memStream = new MemoryStream();
await stream.CopyToAsync(memStream, cancellationToken);
StreamContent fileContent = new(memStream);
fileContent.Headers.ContentType = new MediaTypeHeaderValue("image/jpeg");
content.Add(fileContent, "images", fileName);
responseMessage = await _httpClient.PostAsync(url, content, cancellationToken);
}
Note: stream is the stream of the file, in this case it comes from an ASP.NET Core API controller usingIFormFile.OpenReadStream() but I also tried opening the file directly via
new FileStream("path", FileMode.Open, FileAccess.Read)
In the Debugger content.ReadAsStringAsync() resolves to the following
--abcdef1234567890
Content-Type: text/plain; charset=utf-8
Content-Disposition: form-data; name=organs
flower
--abcdef1234567890
Content-Type: image/jpeg
Content-Disposition: form-data; name=images; filename=test-flower.jpeg; filename*=utf-8''test-flower.jpeg
--abcdef1234567890--
which is looking absolutely fine for me, so my guess is, that somehow the file binary data may be corrupt in the content or something?
When I use the above for VS Code rest client with the same file it works and I get a successful response from the PlantNet API.
(Background: I am using .NET 6 on Fedora Linux)

Ok I solved it by removing the copy to the memory stream. This was needed as at first for debugging I opened the file directly and received exceptions if I didn't do it.
The code that is working for me is
HttpResponseMessage responseMessage;
using (MultipartFormDataContent content = new("abcdef1234567890"))
{
content.Add(new StringContent("flower"), "organs");
StreamContent fileContent = new(stream);
fileContent.Headers.ContentType = new MediaTypeHeaderValue("image/jpeg");
fileContent.Headers.ContentDisposition = new ContentDispositionHeaderValue("form-data")
{
FileName = fileName,
Name = "images"
};
content.Add(fileContent, "images", fileName);
responseMessage = await _httpClient.PostAsync(url, content, cancellationToken);
}

Twilio download recording to file throwing error on JsonConvert.DeserializeObject()

I'm trying to download my recordings on Twilio to a file on my servers local file system (so I can send them to another storage location), but following the code that I've found is throwing an error on the JsonConvert.DeserializeObject() call.
The code that I found is here (called "Retrieve the actual recording media"): https://www.twilio.com/docs/video/api/recordings-resource#filter-by-participant-sid
Here's the code:
static void Main(string[] args)
{
// Find your Account SID and Auth Token at twilio.com/console
const string apiKeySid = "SKXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX";
const string apiKeySecret = "your_api_key_secret";
const string recordingSid = "RTXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX";
const string uri = $"https://video.twilio.com/v1/Recordings/{recordingSid}/Media";
var request = (HttpWebRequest)WebRequest.Create(uri);
request.Headers.Add("Authorization", "Basic " + Convert.ToBase64String(Encoding.ASCII.GetBytes(apiKeySid + ":" + apiKeySecret)));
request.AllowAutoRedirect = false;
string responseBody = new StreamReader(request.GetResponse().GetResponseStream()).ReadToEnd();
var mediaLocation = JsonConvert.DeserializeObject<Dictionary<string, string>>(responseBody)["redirect_to"];
Console.WriteLine(mediaLocation);
new WebClient().DownloadFile(mediaLocation, $"{recordingSid}.out");
}
And here's my version:
var twilioRecordingUri = $"https://api.twilio.com/2010-04-01/Accounts/{recording.AccountSid}/Recordings/{recording.Sid}.mp3?Download=true";
var request = (HttpWebRequest)WebRequest.Create(new Uri(twilioRecordingUri));
request.Headers.Add("Authorization", "Basic " + Convert.ToBase64String(Encoding.ASCII.GetBytes($"{apiKeySid}:{apiKeySecret}")));
request.ContentType = "audio/mpeg";
//request.Accept = "audio/mpeg";
request.AllowAutoRedirect = false;
var responseBody = new StreamReader(request.GetResponse().GetResponseStream()).ReadToEnd();
var deserialized = JsonConvert.DeserializeObject<Dictionary<string, string>>(responseBody);
var mediaLocation = deserialized["redirect_to"];
new WebClient().DownloadFile(mediaLocation, $"{recording.Sid}.out");
But executing that code, it fails on the JsonConvert.Deserialize(), like I mentioned; it throws this generic Json error:
Unexpected character encountered while parsing value: �. Path '', line
0, position 0.
Hovering over my "responseBody" variable does show that it's a really long string of funky characters.
My thought was that I should be adding either the "Accept" or "Content-type" to "audio/mpeg" since that's the type of file that I'm trying to retrieve. But when checking Dev Tools at both the request and response headers, neither the Accept or Content-type ever get my audio/mpeg setting that I just specified.
What's wrong with this code here?
Edit: for anyone that noticed the download URL is different from Twilio's example, I found this page that had the updated URL: How do I get a call recording url in twilio when programming in PHP?

I'm only posting this answer to show what the "working" version looks like. It was #Grungondola 's answer that prompted me. Thanks goes to him (as well as the accepted answer).
private async Task DownloadRecording(RecordingResource recording, string fileName)
{
if (string.IsNullOrEmpty(fileName))
throw new ArgumentNullException("fileName is required when downloading a recording.");
var twilioRecordingUri = $"https://api.twilio.com/2010-04-01/Accounts/{recording.AccountSid}/Recordings/{recording.Sid}.mp3?Download=false";
var request = (HttpWebRequest)WebRequest.Create(new Uri(twilioRecordingUri));
request.Headers.Add("Authorization", "Basic " + Convert.ToBase64String(Encoding.ASCII.GetBytes($"{apiKeySid}:{apiKeySecret}")));
request.ContentType = "audio/mpeg";
request.AllowAutoRedirect = false;
var stream = request.GetResponse().GetResponseStream();
var virtualPath = HttpContext.Server.MapPath(fileName);
var fileStream = new FileStream(virtualPath, FileMode.Create);
await stream.CopyToAsync(fileStream);
fileStream.Close();
}

It looks like the call you're trying to make is to download an .mp3 file and not a Dictionary<string, string>, so it's likely hitting an error when attempting to deserialize a string into a type that it doesn't match. What you're probably seeing as a result is a Base64 string, especially based on your description. Without seeing at least a sample of the data, I can't know for sure, but I'd guess that you're downloading the raw .mp3 file instead of the file information with location (redirect_to).
If the result is a pure Base64 string, you should be able to convert it to an array of bytes and write that directly to a file with whatever filename you want. That should get you the mp3 file that you want.

HttpClient wrong encoding in Content-Disposition

I am POST-ing an image with HttpClient and it works well for files with Latin names, but as soon as a name contains any non-ASCII characters it gets transformed to a sequence of question marks. If I create an html form and use a browser to post the file, the file name is sent in UTF8 and the target server perfectly accepts it.
using (var client = new HttpClient())
{
var streamContent = new StreamContent(someImageFileStream);
streamContent.Headers.Add(
"Content-Disposition",
"form-data; name=\"image\"; filename=\"Тест.jpg\"");
var content = new MultipartFormDataContent();
content.Add(streamContent);
await client.PostAsync("http://localhost.fiddler/", content);
}
This produces the following request:
POST http://localhost/ HTTP/1.1
Content-Type: multipart/form-data; boundary="e6fe89be-e652-4fe3-8859-8c7a339c5550"
Host: localhost
Content-Length: 10556
--e6fe89be-e652-4fe3-8859-8c7a339c5550
Content-Disposition: form-data; name="image"; filename="????.jpg"
...here goes the contents of the file...
I understand that HttpClient might work according to some standard, but anyway, is there any workaround?
UPDATE: The external API doesn't want to accept the format filename*=utf-8''Тест.jpg, it expects filename="Тест.jpg".

This is another way to workaround the limitation of HttpClient without tampering with internal fields. Inspired by this answer.
using (var client = new HttpClient())
{
var streamContent = new StreamContent(someImageFileStream);
streamContent.Headers.Add("Content-Disposition",
new string(Encoding.UTF8.GetBytes("form-data; name=\"image\"; filename=\"Тест.jpg\"").
Select(b => (char)b).ToArray()));
var content = new MultipartFormDataContent();
content.Add(streamContent);
await client.PostAsync("http://localhost.fiddler/", content);
}
I confirm that even .net core 2.2 doesn't have proper support for uploading files whose names contain non-ASCII characters. HttpClient does work according to some standard but Java servers don't care about that standard and expect UTF-8 formatted headers.

OK, I've found a way to force MultipartFormDataContent to forget the ancient RFCs and use UTF8 instead. The trick is to use reflection to overwrite the DefaultHttpEncoding defined in the internal static class HttpRuleParser.
typeof(HttpClient)
.Assembly
.GetType("System.Net.Http.HttpRuleParser")
.GetField("DefaultHttpEncoding", BindingFlags.Static | BindingFlags.NonPublic)
.SetValue(null, System.Text.Encoding.UTF8);
Not sure which bad consequences that might cause, but I suppose there are none.

Instead of adding a header that you built yourself, use the .NET library:
streamContent.Headers.ContentDisposition =
new System.Net.Http.Headers.ContentDispositionHeaderValue("form-data") {
Name = "image",
FileName = "Тест.jpg" };
That creates the header per the web docs and RFC5987.
Content-Disposition: form-data; name=image; filename="=?utf-8?B?0KLQtdGB0YIuanBn?="

If it helps, you can also remove the "filename*"
//It deletes filename* parametr
foreach (var content in multipartContent) {
var headerContent = content.Headers.ContentDisposition.Parameters.Where(x => x.Name == "filename*").SingleOrDefault();
if(headerContent != null)
content.Headers.ContentDisposition.Parameters.Remove(headerContent);
}

How to read http response of multi-part stream in c#

I need to receive response from a http server using c#.The response is a multi-part stream; the header would contain
Content-Type: multipart/x-mixed-replace; boundary=userdata
The body of response would be similar to below:
--userdata
Content-type: text/plain
Content-length: <length-of-content>
UserId: <user-Id>
ParentId: <parent-Id>
ParentName: <parent-name>
Time: <time>
--userdata
Content-type: text/plain
Content-length: <length-of-content>
UserId: <user-Id>
ParentId: <parent-Id>
ParentName: <parent-name>
Time: <time>
I need to recieve the response and save these information continuously in
List<UserData>
which contains the list of UserData class having information of user.
Http response url is like
http://username:password#userIp:port-number/transactionStream
I have written following code but no result, Please help:
NetworkCredential networkCredential = new NetworkCredential(this.UserName, this.Password);
string requestingURL = "http://" + this.UserName + ":" + this.Password + "#" +this.UserIp+ ":" + this.PortNo + "/transactionDataStream";
Uri uri = new Uri(requestingURL);
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uri);
request.Credentials = networkCredential;
HttpWebResponse Answer = (HttpWebResponse)request.GetResponse();
Stream aStream = Answer.GetResponseStream();
StreamReader aStreamReader = new StreamReader(aStream);
string response = aStreamReader.ReadToEnd();

I've used this before and it works well. StreamingMultipartFormDataParser
The Http Multipart Parser does it exactly what it claims on the tin: parses multipart/form-data. This particular parser is well suited to parsing large data from streams as it doesn't attempt to read the entire stream at once and produces a set of streams for file data.

Big files uploading (WebException: The connection was closed unexpectedly)

UPDATED
See post #3 below.
There is a need to upload a file to the web automatically (without browser). Host - Mini File Host v1.2 (if this does matter). Didn't find specific api in documentation, so at first i sniffed browser requests in Firebug as follows :
Params : do
Value : verify
POST /upload.php?do=verify HTTP/1.1
Host: webfile.ukrwest.net
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.2; ru; rv:1.9.0.8) Gecko/2009032609 Firefox/3.0.8 (.NET CLR 4.0.20506)
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: ru,en-us;q=0.7,en;q=0.3
Accept-Encoding: gzip,deflate
Accept-Charset: windows-1251,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Referer: http://filehoster.awardspace.com/index.php
Content-Type: multipart/form-data; boundary=---------------------------27368237179714
Content-Length: 445
-----------------------------27368237179714
Content-Disposition: form-data; name="upfile"; filename="Test.file"
Content-Type: application/octet-stream
12345678901011121314151617sample text
-----------------------------27368237179714
Content-Disposition: form-data; name="descr"
-----------------------------27368237179714
Content-Disposition: form-data; name="pprotect"
-----------------------------27368237179714--
Here we can see parameter, headers, content type and chunks of information (1 - file name and type, 2 - file contents, 3 - additional params - description and password, not necessarily applied).
So i've created a class that emulates such a behaviour step by step : HttpWebRequest on the url, apply needed parameters to request, form request strings with StringBuilder and convert them to byte arrays, read a file using FileStream, putting all that stuff to MemoryStream and then writing it to request (took major part of a code from an article at CodeProject where it uploads a file to Rapidshare host).
Neat and tidy, but... It doesn't seem to work :(. As result it returns initial upload page, not a result page with links i could parse and present to a user...
Here are main methods of an Uploader class :
// Step 1 - request creation
private HttpWebRequest GetWebrequest(string boundary)
{
Uri uri = new Uri("http://filehoster.awardspace.com/index.php?do=verify");
System.Net.HttpWebRequest httpWebRequest = (System.Net.HttpWebRequest)System.Net.WebRequest.Create(uri);
httpWebRequest.CookieContainer = _cookies;
httpWebRequest.ContentType = "multipart/form-data; boundary=" + boundary;
httpWebRequest.UserAgent = "Mozilla/5.0 (Windows; U; Windows NT 5.2; ru; rv:1.9.0.8) Gecko/2009032609 Firefox/3.0.8 (.NET CLR 4.0.20506)";
httpWebRequest.Referer = "http://filehoster.awardspace.com/index.php";
httpWebRequest.Method = "POST";
httpWebRequest.KeepAlive = true;
httpWebRequest.Timeout = -1;
//httpWebRequest.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
httpWebRequest.Headers.Add("Accept-Charset", "windows-1251,utf-8;q=0.7,*;q=0.7");
httpWebRequest.Headers.Add("Accept-Encoding", "gzip,deflate");
httpWebRequest.Headers.Add("Accept-Language", "ru,en-us;q=0.7,en;q=0.3");
//httpWebRequest.AllowAutoRedirect = true;
//httpWebRequest.ProtocolVersion = new Version(1,1);
//httpWebRequest.SendChunked = true;
//httpWebRequest.Headers.Add("Cache-Control", "no-cache");
//httpWebRequest.ServicePoint.Expect100Continue = false;
return httpWebRequest;
}
// Step 2 - first message part (before file contents)
private string GetRequestMessage(string boundary, string FName, string description, string password)
{
System.Text.StringBuilder stringBuilder = new StringBuilder();
stringBuilder.Append("--");
stringBuilder.Append(boundary);
stringBuilder.Append("\r\n");
stringBuilder.Append("Content-Disposition: form-data; name=\"");
stringBuilder.Append("upfile");
stringBuilder.Append("\"; filename=\"");
stringBuilder.Append(FName);
stringBuilder.Append("\"");
stringBuilder.Append("\r\n");
stringBuilder.Append("Content-Type: application/octet-stream");
stringBuilder.Append("\r\n");
return stringBuilder.ToString();
}
// Step 4 - additional request parameters. Step 3 - reading file is in method below
private string GetRequestMessageEnd(string boundary)
{
System.Text.StringBuilder stringBuilder = new StringBuilder();
stringBuilder.Append(boundary);
stringBuilder.Append("\r\n");
stringBuilder.Append("Content-Disposition: form-data; name=\"descr\"");
stringBuilder.Append("\r\n");
stringBuilder.Append("\r\n");
stringBuilder.Append("Default description");
stringBuilder.Append("\r\n");
stringBuilder.Append(boundary);
stringBuilder.Append("\r\n");
stringBuilder.Append("Content-Disposition: form-data; name=\"pprotect\"");
stringBuilder.Append("\r\n");
stringBuilder.Append("\r\n");
stringBuilder.Append("");
stringBuilder.Append("\r\n");
stringBuilder.Append(boundary);
stringBuilder.Append("--");
//stringBuilder.Append("\r\n");
//stringBuilder.Append(boundary);
//stringBuilder.Append("\r\n");
return stringBuilder.ToString();
}
// Main method
public string ProcessUpload(string FilePath, string description, string password)
{
// Chosen file information
FileSystemInfo _file = new FileInfo(FilePath);
// Random boundary generation
DateTime dateTime2 = DateTime.Now;
long l2 = dateTime2.Ticks;
string _generatedBoundary = "----------" + l2.ToString("x");
// Web request creation
System.Net.HttpWebRequest httpWebRequest = GetWebrequest(_generatedBoundary);
// Main app block - form and send request
using (System.IO.FileStream fileStream = new FileStream(_file.FullName, FileMode.Open, FileAccess.Read, FileShare.Read))
{
byte[] bArr1 = Encoding.ASCII.GetBytes("\r\n--" + _generatedBoundary + "\r\n");
// Generating pre-content post message
string firstPostMessagePart = GetRequestMessage(_generatedBoundary, _file.Name, description, password);
// Writing first part of request
byte[] bArr2 = Encoding.UTF8.GetBytes(firstPostMessagePart);
Stream memStream = new MemoryStream();
memStream.Write(bArr1, 0, bArr1.Length);
memStream.Write(bArr2, 0, bArr2.Length);
// Writing file
byte[] buffer = new byte[1024];
int bytesRead = 0;
while ((bytesRead = fileStream.Read(buffer, 0, buffer.Length)) != 0)
{
memStream.Write(buffer, 0, bytesRead);
}
// Generating end of a post message
string secondPostMessagePart = GetRequestMessageEnd(_generatedBoundary);
byte[] bArr3 = Encoding.UTF8.GetBytes(secondPostMessagePart);
memStream.Write(bArr3, 0, bArr3.Length);
// Preparing to send
httpWebRequest.ContentLength = memStream.Length;
fileStream.Close();
Stream requestStream = httpWebRequest.GetRequestStream();
memStream.Position = 0;
byte[] tempBuffer = new byte[memStream.Length];
memStream.Read(tempBuffer, 0, tempBuffer.Length);
memStream.Close();
// Sending request
requestStream.Write(tempBuffer, 0, tempBuffer.Length);
requestStream.Close();
}
// Delay (?)
System.Threading.Thread.Sleep(5000);
// Getting response
string strResponse = "";
using (Stream stream = httpWebRequest.GetResponse().GetResponseStream())
using (StreamReader streamReader = new StreamReader(stream/*, Encoding.GetEncoding(1251)*/))
{
strResponse = streamReader.ReadToEnd();
}
return strResponse;
}
Plays with ProtocolVersion (1.0, 1.1), AllowAutoRedirect (true/false), even known ServicePoint.Expect100Continue (false) didn't fix an issue. Even a 5sec timeout before getting response (thought in case of a big file it doesn't uploads so quick) didn't help.
Content type "octet-stream" was chosen by purpose to upload any file (could use some switch for most popular jpg/zip/rar/doc etc., but that one seems universal). Boundary is generated randomly from timer ticks, not a big deal. What else? :/
I could give up and forget this, but i feel i'm pretty close to solve and then forget about it :P.
In case you need the whole application to run and debug - here it is (70kb, zipped C# 2.0 VS2k8 solution, no ads, no viruses) :
#Mediafire
#FileQube
#FileDropper

Update : nope, there is no redirect.
screenshot
Read RFC2388 few times, rewrote the code and it finally worked (i guess the trouble was in utf-read trailing boundary instead of correct 7 bit ascii). Hooray? Nope :(. Only small files are transfered, big ones throwing "The connection was closed unexpectedly".
System.Net.WebException was unhandled by user code
Message="The underlying connection was closed: The connection was closed unexpectedly."
Source="Uploader"
StackTrace:
at Uploader.Upload.ProcessUpload(String FilePath, String description, String password) in F:\MyDocuments\Visual Studio 2008\Projects\Uploader\Uploader.cs:line 96
at Uploader.Form1.backgroundWorker1_DoWork(Object sender, DoWorkEventArgs e) in F:\MyDocuments\Visual Studio 2008\Projects\Uploader\Form1.cs:line 45
at System.ComponentModel.BackgroundWorker.WorkerThreadStart(Object argument)
I know that's a bug with .net stack and few solutions exists :
increase both Timeout and ReadWriteTimeout of a request
assign request.KeepAlive = false and System.Net.ServicePointManager.Expect100Continue = false
set ProtocolVersion to 1.0
But neither one of them nor all of them altogether help in my case. Any ideas?
EDIT - Source code:
// .. request created, required params applied
httpWebRequest.ProtocolVersion = HttpVersion.Version10; // fix 1
httpWebRequest.KeepAlive = false; // fix 2
httpWebRequest.Timeout = 1000000000; // fix 3
httpWebRequest.ReadWriteTimeout = 1000000000; // fix 4
// .. request processed, data written to request stream
string strResponse = "";
try
{
using (WebResponse httpResponse = httpWebRequest.GetResponse()) // error here
{
using (Stream responseStream = httpResponse.GetResponseStream())
{
using (StreamReader streamReader = new StreamReader(responseStream))
{
strResponse = streamReader.ReadToEnd();
}
}
}
}
catch (WebException exception)
{
throw exception;
}

"As result it returns initial upload page, not a result page with links i could parse and present to a user..."
Maybe that's just the behaviour of the upload functionality: that after the upload has finished, you can upload another file?
I think you have to call another url for the "browse for file"-page (I suppose that's the one you need).
Edit: Actually, if the server sends a "redirect" (http 3xx), that's something the browser has to handle, so if you're working with your own client application in stead of a browser, you'll have to implement redirection yourself. Here the rfc for more information.

Try setting the maxRequestLength property of the httpRuntime element in the Web.config.

In my case, duplicate filename causing the issue as well. I save the file's settings in an xml file but the name setting is duplicating each other.
<field name="StillImage">
<prefix>isp_main_</prefix>
<suffix>308</suffix>
<width>1080</width>
<height>1080</height>
</field>
<field name="ThumbnailImage">
<prefix>isp_thumb_</prefix> // pay attention to this
<suffix>308</suffix>
<width>506</width>
<height>506</height>
</field>
<field name="Logo">
<prefix>isp_thumb_</prefix> // and this
<suffix>306</suffix>
<width>506</width>
<height>506</height>
</field>
And, in the other case I had, the issue is in the file length. Please do check the allowed file size on your server. In your script just do check this part :
dataStream.Write(filesBytesArray, 0, filesBytesArray.Length);
dataStream.Close();
And if you dont know, just limit the file uploaded size in your frontend section ie. HTML <input type="file"> upload element, this is good reference for limiting file size and other filter.