We Keep Coding

How to send unencoded form data with C# HttpClient

I'm trying to "repurpose" a third-party API used by a desktop application. I've found that the below code gets me very close to matching the packets sent by the app:
var formData = new FormUrlEncodedContent(new[]
{
new KeyValuePair<string, string>(JsonConvert.SerializeObject(myPayload), "")
});
var response = Client.PostAsync(myURL, formData).Result;
var json = response.Content.ReadAsStringAsync().Result;
This gets me almost exactly the same payload sent by the application, except it encodes the data (I know, "encoded" is right there in the name). I need to get the exact same request but without the data being encoded, but I can't quite find the right object(s) to pull it off. How do I keep this payload from being URL encoded?
Edit:
This is a login request I pulled from Wireshark emanating from the application:
POST /Login HTTP/1.1
Content-Type: application/x-www-form-urlencoded
Host: 1.1.1.1
Content-Length: 161
Expect: 100-continue
Connection: Close
{"username":"myuser","auth-id":"0a0a140f81a2ce0c303386e93cec41bf04660c22a881be9a"}
This is what the above will generate:
POST /Login HTTP/1.1
Expect: 100-continue
Connection: Close
Content-Type: application/x-www-form-urlencoded
Content-Length: 221
Host: 1.1.1.1
%7B%22user-name%22%3A%22myuser%22%2C%22auth-id%22%3A%220a0a140f81a2ce0c303386e93cec41bf04660c22a881be9a%22%7D=
I've edited them for brevity so the Content-Length is wrong. I realize it might not be the best way to send this data, but I have no control over how it's consumed.

Since you're actually trying to send JSON, I think you need to wrap the JSON in a StringContent object rather than a FormUrlEncoded object. Form-encoded data and JSON data are two different ways of formatting a payload (another commonly used format would be XML, for example). Using them both together doesn't make any sense.
I think something like this should work:
var content = new StringContent(JsonConvert.SerializeObject(myPayload), Encoding.UTF8, "application/json");
var response = Client.PostAsync(myURL, content).Result;
var json = response.Content.ReadAsStringAsync().Result;
(P.S. the Content-Type: application/x-www-form-urlencoded header sent by the application appears to be misleading, since the request body clearly contains JSON. Presumably the receiving server is tolerant of this nonsense, or just ignores it because it's always expecting JSON.)

Return json representations of data from WebAPI without strongly-typed IEnumerable

Having read a ton of different solutions and trying out different approaches (here, here, here, here, here), I am getting different (and unexpected) results from an API controller.
I need to return a datatable expressed as JSON, via a web api controller:
[IdentityBasicAuthentication]
[Authorize]
[RequireHttps]
[Route("Reports/Report1")]
public HttpResponseMessage Get()
{
DataTable dt = MyDAL.GetDataTable();
if(someValidationFailed){
Request.CreateResponse(HttpStatusCode.BadRequest, "Friendly error here");
}
return Request.CreateResponse(HttpStatusCode.OK, Newtonsoft.Json.JsonConvert.SerializeObject(dt));
}
In Fiddler, the output appears to be encoded into an escaped string:
HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
"[{\"Site\":\"Headquarters\",\"Department\":\"HR\",\"FirstName\":\"Bob\",\"MiddleName\":null,\"Surname\":\"Fern\",\"EmployeeNumber\":\"444\"},
{\"Site\":\"Headquarters\",\"Department\":\"HR\",\"FirstName\":\"Alice\",\"MiddleName\":null,\"Surname\":\"Smith\",\"EmployeeNumber\":\"769\"}]"
However, the goal is to output plain Json (or XML if the client requests it):
HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
[{"Site":"Headquarters","Department":"HR","FirstName":"Bob","MiddleName":null,"Surname":"Fern","EmployeeNumber":"444"},
{"Site":"Headquarters","Department":"HR","FirstName":"Alice","MiddleName":null,"Surname":"Smith","EmployeeNumber":"769"}]
Is there any way to do this without creating an IEnumerable<Person> object? The reason is that this class is specific to this one method. I also don't want to use a StringBuilder, because the DataTable may be extremely large and I've seen memory exceptions thrown previously.
I'm new to WebAPI, so the answer may be really simple, but a code sample I can ask further questions on would be very helpful.

Web API handles serialization for you, so you do not need to call JsonConvert.SerializeObject. That is why you are getting an escaped string as your output value. Just pass the datatable directly to CreateResponse. Web API will turn it into JSON or XML for you depending on what was sent in the Accept header of the request. (It uses Json.Net under the covers.)
return Request.CreateResponse(HttpStatusCode.OK, dt);

Use anonymous type. JSON support serialization of anonymous types.
var persons = datatable.AsEnumerable()
.Select(datarow => new
{
Site = datarow.Field<string>("Site")
});
return Request.CreateResponse(HttpStatusCode.OK,
Newtonsoft.Json.JsonConvert.SerializeObject(persons));

Override the default Content-Type for responses in NancyFx

I'm writing a REST API with NancyFx. I often got code like this:
Post["/something"] = _ => {
// ... some code
if (success)
return HttpStatusCode.OK;
else
return someErrorObject;
};
The client always assumes application/json as the content type of all responses. It actually sets Accept: application/json in the request. Responses without application/json are errors regardless of the actual body. It simply checks the content type and aborts if it doesn't match json. I can't change this behaviour.
Now I face the problem that by simply returning HttpStatusCode.OK Nancy sets Content-Type: text/html but as said the client assumes application/json and fails with an error even the body is empty.
I was able to force the content type like this:
return Negotiate
.WithContentType("application/json")
.WithStatusCode(HttpStatusCode.OK);
I don't want to repeat this code everywhere. Of course I could abstract that in a single function but I'm looking for a more elegant solution.
Is there a way to override the default Content-Type of respones so that return HttpStatusCode.OK sets my Content-Type to application/json?

Based on the assumption of wanting to return all responses as JSON you will need a custom bootstrapper. You can enhance this further if you like by using an insert as opposed to clearing the response processors thus the fallback of using the XML processor etc is possible.
This will get automatically picked up by Nancy by the way no additional configuration required.
public class Bootstrap : DefaultNancyBootstrapper
{
protected override NancyInternalConfiguration InternalConfiguration
{
get
{
return NancyInternalConfiguration.WithOverrides(
(c) =>
{
c.ResponseProcessors.Clear();
c.ResponseProcessors.Add(typeof(JsonProcessor));
});
}
}
}

Telling RestSharp *not* to add a specific HTTP header

I am trying to call a REST service from a C# ASP.NET 4.0 application using RestSharp.
It's a fairly straightforward POST call to a https:// address; my code is something like this (CheckStatusRequest is a plain simple DTO with about four or five string and int properties - nothing fancy):
public CheckStatusResponse CheckStatus(CheckStatusRequest request) {
// set up RestClient
RestClient client = new RestClient();
string uri = "https://.......";
// create the request (see below)
IRestRequest restRequest = CreateRequestWithHeaders(url, Method.POST);
// add the body to the request
restRequest.AddBody(request);
// execute call
var restResponse = _restClient.Execute<CheckStatusResponse>(restRequest);
}
// set up request
private IRestRequest CreateRequestWithHeaders(string uri, Method method) {
// define request
RestRequest request = new RestRequest(uri, method);
// add two required HTTP headers
request.AddHeader("Accept", "application/json");
request.AddHeader("Content-Type", "application/json");
// define JSON as my format
request.RequestFormat = DataFormat.Json;
// attach the JSON.NET serializer for RestSharp
request.JsonSerializer = new RestSharpJsonNetSerializer();
return request;
}
The problem I'm having when I send these requests through Fiddler to see what's going on is that my request suddenly gets a third and unwanted HTTP header:
POST https://-some-url- HTTP/1.1
Accept: application/json
User-Agent: RestSharp/104.4.0.0
Content-Type: application/json
Host: **********.com
Content-Length: 226
Accept-Encoding: gzip, deflate <<<=== This one here is UNWANTED!
Connection: Keep-Alive
I suddenly have that Accept-Encoding HTTP header, which I never specified (and which I don't want to have in there). And now my response is no longer proper JSON (which I'm able to parse), but suddenly I get back gzipped binary data instead (which doesn't do real well when trying to JSON-deserialize)....
How can I get rid of that third unwanted HTTP header?
I tried to set it to something else - whatever I enter just gets appended to those settings
I tried to somehow "clear" that HTTP header - without any success
I tried finding a property on the RestClient or the RestRequest classes to specify "do not use GZip"

Looking at the sources (Http.Sync.cs and Http.Async.cs) of RestSharp you can see that these values are hardcoded:
webRequest.AutomaticDecompression =
DecompressionMethods.Deflate | DecompressionMethods.GZip | DecompressionMethods.None;
There is also an open issue that describes this problem. It was opened August 2014 but still not solved. I think you can leave a comment there and maybe they will pay attention.

HttpClient.SendAsync not sending request body

I am using the ASP.NET Web API Client Libraries for .NET 4.0 (Microsoft.AspNet.WebApi.Client version 4.0.30506.0).
I need to send an HTTP DELETE with a request body. I have coded it as follows:
using (var client = new HttpClient())
{
client.BaseAddress = Uri;
client.DefaultRequestHeaders.Accept.Clear();
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
// I would normally use httpClient.DeleteAsync but I can't because I need to set content on the request.
// For this reason I use httpClient.SendAsync where I can both specify the HTTP DELETE with a request body.
var request = new HttpRequestMessage(HttpMethod.Delete, string.Format("myresource/{0}", sessionId))
{
var data = new Dictionary<string, object> {{"some-key", "some-value"}};
Content = new ObjectContent<IDictionary<string, object>>(data, new JsonMediaTypeFormatter())
};
var response = await client.SendAsync(request);
// code elided
}
Per Fiddler, the request body is never serialized:
DELETE http://localhost:8888/myApp/sessions/blabla123 HTTP/1.1
Accept: application/json
Content-Type: application/json; charset=utf-8
Host: localhost:8888
Content-Length: 38
Expect: 100-continue
The response from the server:
HTTP/1.1 408 Request body incomplete
Date: Sun, 10 Aug 2014 17:55:17 GMT
Content-Type: text/html; charset=UTF-8
Connection: close
Cache-Control: no-cache, must-revalidate
Timestamp: 13:55:17.256
The request body did not contain the specified number of bytes. Got 0, expected 38
I have tried a number of workarounds, including changing the type being serialized to something else, doing the serialization myself with JsonSerialize, changing the HTTP DELETE to PUT, etc...
Nothing worked. Any help would be much appreciated.

I resolved the issue, though it does not make sense. I noticed that if I changed my call to HTTP PUT or POST, it still failed to serialize the Content as a request body. That was rather strange as previous PUTs and POSTs were successful. After doing a ton of debugging into framework libraries (using Reflector), I finally got to the only thing left that was "different."
I am using NUnit 2.6.2. The structure of my test is:
[Test]
async public void Test()
{
// successful HTTP POST and PUT calls here
// successful HTTP DELETE with request body here (after
// moving it from the TearDown below)
}
[TearDown]
async public void TerminateSession()
{
// failed HTTP DELETE with request body here
}
Why does this fail in the TearDown but not in the Test itself? I have no idea. Is something going on with the TearDown attribute or with the use of the async keyword (since I await async calls)?
I am not sure what it is causing this behavior, but I do know now that I can submit an HTTP DELETE with a request body (as outlined in my code sample in the question).
Another solution that worked is as follows:
[Test]
async public void Test()
{
// create and use an HttpClient here, doing POSTs, PUTs, and GETs
}
// Notice the removal of the async keyword since now using Wait() in method body
[TearDown]
public void TerminateSession()
{
// create and use an HttpClient here and use Wait().
httpClient.SendAsync(httpRequestMessage).Wait();
}

I know it's never quite that helpful to say, "don't do it that way", but in this case I think it makes sense to split the calls into a DELETE followed or preceeded by a POST or PUT.
The HTTP RFC doesn't explicitly opine on the matter, so technically it means that we can. The other question, however, is should we do it.
In cases such as this I would look for other implementations to see what is the de facto standard. As you've found in the .net implementation, it appears that the designers did not expect to send a body with the DELETE call. So, let's look at another popular (and very different impl) Python Requests:
>>> r = requests.delete(url=url, auth=auth)
>>> r.status_code
204
>>> r.headers['status']
'204 No Content'
No body here other. So, if the spec authors didn't mention it, and popular implementations assume that there's no body, then the principle of least surprise means we shouldn't do it either.
So, if you can change the API, it will be easier on clients of the API to split into two calls. Otherwise, you'll likely have to resort to custom hackery to cram the body into a DELETE call.
The good news is that you've likely found a bug in the .net framework, which is an achievement in and of itself. Clients advertising a non-zero Content-Length without actually sending it are broken.

In case anybody else runs into this, one thing I've noticed that can cause this is if you set a header with a newline in it.
We had an encrypted OAuth token, which gets decrypted at runtime and set as the OAuth header on the app. The newline was encrypted into the token, so it was not obvious from looking at the configs or anything that it was there, but if you do:
var message = new HttpRequestMessage(HttpMethod.Post, "https://example.com");
message.Headers.ContentType = new MediaTypeHeaderValue("application/json");
message.Content = new StringContent("{ \"someKey\": \"someValue\" }", Encoding.UTF8);
// note the trailing newline
message.Headers.Authorization = new AuthenticationHeaderValue("OAuth", "my auth token\n");
var response = await httpClient.SendAsync(request);
The HTTP request will be sent, but the content will not be sent with it. There are no exceptions thrown when this happens and if you inspect the HttpRequestMessage, the content will appear to be there, but it does not actually get sent over the wire.
This happens in .NET 5 on Windows and Linux, I haven't tested it on other framework versions/platforms.