We Keep Coding

How do you set the Content-Type header for an HttpClient request with MultipartFormDataContent?

I looked at the MS Source code according to their interpretation the HttpClient itself does not have "Content-Type" only the content should have content-type. Seems logical except when you're dealing with MultipartFormDataContent.
MultipartFormDataContent completely ignores the following code:
string boundary = "--" + GenerateRandomString();
using (var content = new MultipartFormDataContent(boundary))
{
content.Headers.ContentType = new MediaTypeHeaderValue($"multipart/form-data");
content.Headers.ContentType.Parameters.Add(new NameValueHeaderValue("boundry", boundary));
...
}
No "Content-Type" is present in the request.
And also ignores:
string boundary = "--" + GenerateRandomString();
using (var content = new MultipartFormDataContent(boundary))
{
content.Headers.Remove("Content-Type");
content.Headers.TryAddWithoutValidation("Content-Type", "multipart/form-data; boundary=" + boundary);
...
}
Attemting to set it at on the HttpClient
client.DefaultRequestHeaders.TryAddWithoutValidation("Content-Type", "multipart/form-data; boundary=" + boundary);
throws the following error:
System.InvalidOperationException: 'Misused header name. Make sure request headers are used with HttpRequestMessage, response headers with HttpResponseMessage, and content headers with HttpContent objects.'
I can find plenty of examples of how to do this using StringContent but none with MultipartFormDataContent. MultipartFormDataContent allows setting the Content-Type and Content-Disposition with each field, I need this more at the client level. I need a header that looks something like this:
accept: application/json
Content-Type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW
Authorization: Basic ZXlKbGRDSTZJakUxTWpZNU9UTXpOTnpkMjl5WkNJNklqa3haRGxtWkdKa1lUazRaVEJqWmpsalpUaGxNV1V3TXpOalxuWmpCbE1tVXhJaXdpZFhObGNpSTZJbUZrYldsdUluMD1cbjo=
Cache-Control: no-cache
Many non-Microsoft APIs require the "boundary" tag so it can distinguish the individual fields of data being sent. The validation here on the request seems a little over the top. Even TryAddWithoutValidation doesn't work (maybe a bug?). I realize that it may be possible to interpret RFC7578 in a way that says it shouldn't be required but flat out not allowing it doesn't seem right to me either. Anyone else ever run into this issue and solve it.

Initially I thought this was an HttpClient bug. I added logging to capture the request and the response. That logging was missing headers which lead me to believe that the missing "multi-part/form-data" content header was the issue and the reason the API I'm using kept telling me it couldn't find a required field. It turns out to be an issue with how the API handles the data it's sent when it's multi-part/form-data. After comparing both my HttpWebRequest and the HttpClient request in fiddler I discovered the following difference in the data being sent:
HttpWebRequest
----vekhftkcthxr
Content-Disposition: form-data; name="name";
d30-20180524
HttpClient
----bcgifxyjkmkw
Content-Type: text/plain; charset=utf-8
Content-Disposition: form-data; name=name
d30-20180524
I build the HttpWebRequest manually so I included the quotes and ending semi-colons. The HttpClient request is built for me and does not include the extra quotes and semi-colon. So the API I am using does not play well with request being generated by the HttpClient even though the request is technically correct.
Thanks to Panagiotis Kanavos for showing me my error.

Telling RestSharp *not* to add a specific HTTP header

I am trying to call a REST service from a C# ASP.NET 4.0 application using RestSharp.
It's a fairly straightforward POST call to a https:// address; my code is something like this (CheckStatusRequest is a plain simple DTO with about four or five string and int properties - nothing fancy):
public CheckStatusResponse CheckStatus(CheckStatusRequest request) {
// set up RestClient
RestClient client = new RestClient();
string uri = "https://.......";
// create the request (see below)
IRestRequest restRequest = CreateRequestWithHeaders(url, Method.POST);
// add the body to the request
restRequest.AddBody(request);
// execute call
var restResponse = _restClient.Execute<CheckStatusResponse>(restRequest);
}
// set up request
private IRestRequest CreateRequestWithHeaders(string uri, Method method) {
// define request
RestRequest request = new RestRequest(uri, method);
// add two required HTTP headers
request.AddHeader("Accept", "application/json");
request.AddHeader("Content-Type", "application/json");
// define JSON as my format
request.RequestFormat = DataFormat.Json;
// attach the JSON.NET serializer for RestSharp
request.JsonSerializer = new RestSharpJsonNetSerializer();
return request;
}
The problem I'm having when I send these requests through Fiddler to see what's going on is that my request suddenly gets a third and unwanted HTTP header:
POST https://-some-url- HTTP/1.1
Accept: application/json
User-Agent: RestSharp/104.4.0.0
Content-Type: application/json
Host: **********.com
Content-Length: 226
Accept-Encoding: gzip, deflate <<<=== This one here is UNWANTED!
Connection: Keep-Alive
I suddenly have that Accept-Encoding HTTP header, which I never specified (and which I don't want to have in there). And now my response is no longer proper JSON (which I'm able to parse), but suddenly I get back gzipped binary data instead (which doesn't do real well when trying to JSON-deserialize)....
How can I get rid of that third unwanted HTTP header?
I tried to set it to something else - whatever I enter just gets appended to those settings
I tried to somehow "clear" that HTTP header - without any success
I tried finding a property on the RestClient or the RestRequest classes to specify "do not use GZip"

Looking at the sources (Http.Sync.cs and Http.Async.cs) of RestSharp you can see that these values are hardcoded:
webRequest.AutomaticDecompression =
DecompressionMethods.Deflate | DecompressionMethods.GZip | DecompressionMethods.None;
There is also an open issue that describes this problem. It was opened August 2014 but still not solved. I think you can leave a comment there and maybe they will pay attention.

HttpClient.SendAsync not sending request body

I am using the ASP.NET Web API Client Libraries for .NET 4.0 (Microsoft.AspNet.WebApi.Client version 4.0.30506.0).
I need to send an HTTP DELETE with a request body. I have coded it as follows:
using (var client = new HttpClient())
{
client.BaseAddress = Uri;
client.DefaultRequestHeaders.Accept.Clear();
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
// I would normally use httpClient.DeleteAsync but I can't because I need to set content on the request.
// For this reason I use httpClient.SendAsync where I can both specify the HTTP DELETE with a request body.
var request = new HttpRequestMessage(HttpMethod.Delete, string.Format("myresource/{0}", sessionId))
{
var data = new Dictionary<string, object> {{"some-key", "some-value"}};
Content = new ObjectContent<IDictionary<string, object>>(data, new JsonMediaTypeFormatter())
};
var response = await client.SendAsync(request);
// code elided
}
Per Fiddler, the request body is never serialized:
DELETE http://localhost:8888/myApp/sessions/blabla123 HTTP/1.1
Accept: application/json
Content-Type: application/json; charset=utf-8
Host: localhost:8888
Content-Length: 38
Expect: 100-continue
The response from the server:
HTTP/1.1 408 Request body incomplete
Date: Sun, 10 Aug 2014 17:55:17 GMT
Content-Type: text/html; charset=UTF-8
Connection: close
Cache-Control: no-cache, must-revalidate
Timestamp: 13:55:17.256
The request body did not contain the specified number of bytes. Got 0, expected 38
I have tried a number of workarounds, including changing the type being serialized to something else, doing the serialization myself with JsonSerialize, changing the HTTP DELETE to PUT, etc...
Nothing worked. Any help would be much appreciated.

I resolved the issue, though it does not make sense. I noticed that if I changed my call to HTTP PUT or POST, it still failed to serialize the Content as a request body. That was rather strange as previous PUTs and POSTs were successful. After doing a ton of debugging into framework libraries (using Reflector), I finally got to the only thing left that was "different."
I am using NUnit 2.6.2. The structure of my test is:
[Test]
async public void Test()
{
// successful HTTP POST and PUT calls here
// successful HTTP DELETE with request body here (after
// moving it from the TearDown below)
}
[TearDown]
async public void TerminateSession()
{
// failed HTTP DELETE with request body here
}
Why does this fail in the TearDown but not in the Test itself? I have no idea. Is something going on with the TearDown attribute or with the use of the async keyword (since I await async calls)?
I am not sure what it is causing this behavior, but I do know now that I can submit an HTTP DELETE with a request body (as outlined in my code sample in the question).
Another solution that worked is as follows:
[Test]
async public void Test()
{
// create and use an HttpClient here, doing POSTs, PUTs, and GETs
}
// Notice the removal of the async keyword since now using Wait() in method body
[TearDown]
public void TerminateSession()
{
// create and use an HttpClient here and use Wait().
httpClient.SendAsync(httpRequestMessage).Wait();
}

I know it's never quite that helpful to say, "don't do it that way", but in this case I think it makes sense to split the calls into a DELETE followed or preceeded by a POST or PUT.
The HTTP RFC doesn't explicitly opine on the matter, so technically it means that we can. The other question, however, is should we do it.
In cases such as this I would look for other implementations to see what is the de facto standard. As you've found in the .net implementation, it appears that the designers did not expect to send a body with the DELETE call. So, let's look at another popular (and very different impl) Python Requests:
>>> r = requests.delete(url=url, auth=auth)
>>> r.status_code
204
>>> r.headers['status']
'204 No Content'
No body here other. So, if the spec authors didn't mention it, and popular implementations assume that there's no body, then the principle of least surprise means we shouldn't do it either.
So, if you can change the API, it will be easier on clients of the API to split into two calls. Otherwise, you'll likely have to resort to custom hackery to cram the body into a DELETE call.
The good news is that you've likely found a bug in the .net framework, which is an achievement in and of itself. Clients advertising a non-zero Content-Length without actually sending it are broken.

In case anybody else runs into this, one thing I've noticed that can cause this is if you set a header with a newline in it.
We had an encrypted OAuth token, which gets decrypted at runtime and set as the OAuth header on the app. The newline was encrypted into the token, so it was not obvious from looking at the configs or anything that it was there, but if you do:
var message = new HttpRequestMessage(HttpMethod.Post, "https://example.com");
message.Headers.ContentType = new MediaTypeHeaderValue("application/json");
message.Content = new StringContent("{ \"someKey\": \"someValue\" }", Encoding.UTF8);
// note the trailing newline
message.Headers.Authorization = new AuthenticationHeaderValue("OAuth", "my auth token\n");
var response = await httpClient.SendAsync(request);
The HTTP request will be sent, but the content will not be sent with it. There are no exceptions thrown when this happens and if you inspect the HttpRequestMessage, the content will appear to be there, but it does not actually get sent over the wire.
This happens in .NET 5 on Windows and Linux, I haven't tested it on other framework versions/platforms.

Fetching AccessToken returns "400 - Bad Request" for authenticating cloud service with WNS [duplicate]

PLEASE HELP!! Can't figure out why this simple code given by MSDN doesn't work....
I am using the following code in GetAccessToken() as given in the this MSDN article to get the access token to be used in windows notifications, but it returns "Bad Request 400"
PACKAGE_SECURITY_IDENTIFIER, CLIENT_SECRET are the values obtained when the app was registered with the Windows Store Dashboard
string urlEncodedSid = HttpUtility.UrlEncode(PACKAGE_SECURITY_IDENTIFIER);
string urlEncodedSecret = HttpUtility.UrlEncode(CLIENT_SECRET);
string body = String.Format("grant_type=client_credentials&client_id={0}&client_secret={1}&scope=notify.windows.com", urlEncodedSid, urlEncodedSecret);
string response;
using (WebClient client = new WebClient())
{
client.Headers.Add("Content-Type", "application/x-www-form-urlencoded");
response = client.UploadString("https://login.live.com/accesstoken.srf", body);
}
Any help would be highly appreciated.......

I suspect the problem has to do with either an incorrect package identifier, and / or incorrect client secret.
From the MSDN page Push notification service request and response headers:
RESPONSE DESCRIPTION
--------------- --------------------------
200 OK The request was successful.
400 Bad Request The authentication failed.
Update - I ran the code from the question, using FAKE credentials.
Here is the RAW HTTP request:
POST https://login.live.com/accesstoken.srf HTTP/1.1
Content-Type: application/x-www-form-urlencoded
Host: login.live.com
Content-Length: 88
Expect: 100-continue
Connection: Keep-Alive
grant_type=client_credentials&client_id=test&client_secret=test&scope=notify.windows.com
Here is the server's RAW response:
HTTP/1.1 400 Bad Request
Cache-Control: no-store
Content-Length: 66
Content-Type: application/json
Server: Microsoft-IIS/7.5
X-WLID-Error: 0x80045A78
PPServer: PPV: 30 H: BAYIDSLGN2A055 V: 0
Date: Thu, 21 Mar 2013 12:34:19 GMT
Connection: close
{"error":"invalid_client","error_description":"Invalid client id"}
You will note that the response is a 400. There is also some json that indicates the type of error. In my case, the error is Invalid client id. You probably want to take a look at your response - it will give you an indication of what happened.
I used Fiddler to debug the request/ response.

I found the reason for the error response. In fact it is the wrong PACKAGE_SECURITY_IDENTIFIER and CLIENT_SECRET.
DO NOT type the values. Because associated ASCII values differ. Therefore it is always better to copy and paste directly.
You will probably will get the access token with the simple code snippet.
Cheers

If you're using the new HttpClient API and you're sure you've copied and pasted the SID/secret values correct, you might be experiencing this issue because of encoding, provided you're using the FormUrlEncodedContent class as the content of your POST operation.
Contrary to the examples in the MSDN documentation, you don't want to URL encode the SID and secret values before adding them to the KeyValuePair collection. This is because encoding is implied by the FormUrlEncodedContent class, though I'm not seeing any documentation for this behavior. Hopefully this saves someone some time because I've been wrestling with this all night...

How to parse a raw HTTP response as HttpListenerResponse?

If I have a raw HTTP response as a string:
HTTP/1.1 200 OK
Date: Tue, 11 May 2010 07:28:30 GMT
Expires: -1
Cache-Control: private, max-age=0
Content-Type: text/html; charset=UTF-8
Server: gws
X-XSS-Protection: 1; mode=block
Connection: close
<!doctype html><html>...</html>
Is there an easy way I can parse it into an HttpListenerResponse object? Or at least some kind .NET object so I don't have to work with raw responses.
What I'm doing currently is extracting the header key/value pairs and setting them on the HttpListenerResponse. But some headers can't be set, and then I have to cut out the body of the response and write it to the OutputStream. But the body could be gzipped, or it could be an image, which I can't get to work yet. And some responses contain random characters everywhere, which looks like an encoding problem. It's a lot of trouble.
I'm getting a raw response because I'm using SOCKS to send an HTTP request. The program I'm working on is basically an HTTP proxy that can route requests through a SOCKS proxy, like Privoxy does.

Looks like there's no easy way to convert them. These articles helped:
How to implement custom proxy server?
http://www.jeffcrossman.com/2009/08/27/using-httplistener-to-build-a-proxy
I ended up doing something very similar.

Maybe you want to take a look at the ResponseHeaders property of the Webclient:
WebClient wc = new WebClient();
wc.DownloadStringAsync(new Uri("http://www.foo.com"));
WebHeaderCollection myWebHeaderCollection = myWebClient.ResponseHeaders;
for (int i=0; i < myWebHeaderCollection.Count; i++)
Console.WriteLine ("\t" + myWebHeaderCollection.GetKey(i) +
" = " + myWebHeaderCollection.Get(i));
Please tell me if that isnt what you were searching for.