I am in need to post the following request:
POST http://target-host.com/some/endpoint HTTP/1.1
Content-Type: multipart/form-data; boundary="2e3956ac-de47-4cad-90df-05199a7c1f53"
Accept-Encoding: gzip, deflate
Connection: Keep-Alive
Content-Length: 6971
Host: target-host.com
--2e3956ac-de47-4cad-90df-05199a7c1f53
Content-Disposition: form-data; name="some-label"
value
--2e3956ac-de47-4cad-90df-05199a7c1f53
Content-Disposition: form-data; name="file"; filename="my-filename.txt"
<file contents>
--2e3956ac-de47-4cad-90df-05199a7c1f53--
I can do this really easily with Python requests library as follows:
import requests
with open("some_file", "rb") as f:
byte_string = f.read()
requests.post(
"http://target-host.com/some/endpoint",
data={"some-label": "value"},
files={"file": ("my-filename.txt", byte_string)})
Is there any way to do the same with the Flurl.Http library?
My problem with the documented way of doing it is that it will insert the Content-Type header for each key-value pair and it will insert the filename*=utf-8'' header for the file data. The server I am trying to post the request to, however, does not support this. Also note the double quotes around the name and filename values in the headers.
EDIT: Below is the code I used to make the post request with Flurl.Http:
using System.IO;
using Flurl;
using Flurl.Http;
namespace ConsoleApplication
{
public class Program
{
public static void Main(string[] args)
{
var fs = File.OpenRead("some_file");
var response = "http://target-host.com"
.AppendPathSegment("some/endpoint")
.PostMultipartAsync(mp => mp
.AddString("some-label", "value")
.AddFile("file", fs, "my-filename.txt")
).Result;
}
}
}
According the spec (dated June 2011), sending both filename and filename* is recommended for maximum compatibility:
Many user agent implementations predating this specification do not
understand the "filename*" parameter. Therefore, when both "filename"
and "filename*" are present in a single header field value, recipients
SHOULD pick "filename*" and ignore "filename". This way, senders can
avoid special-casing specific user agents by sending both the more
expressive "filename*" parameter, and the "filename" parameter as
fallback for legacy recipients.
If filename* is actually causing the call to fail, there's a real problem with the server adhering to the HTTP spec. Also, enclosing name and filename in quotes is very non-standard.
That said, Flurl's shortcuts cover the 90% cases, but you can always use the underlying HttpClient APIs to cover unusual cases like this one. In this case I think you need to build up the content manually so you can deal with those Content-Disposition headers:
var mpc = new MultipartContent();
var sc = new StringContent("value");
sc.Headers.Add("Content-Disposition", "form-data; name=\"some-label\"");
mpc.Add(sc);
var fc = new StreamContent(fs);
fc.Headers.Add("Content-Disposition", "form-data; name=\"file\"; filename=\"my-filename.txt\"");
mpc.Add(fc);
Then you can use it with Flurl like this:
var response = await "http://target-host.com"....PostAsync(mpc);
Related
I'm trying to "repurpose" a third-party API used by a desktop application. I've found that the below code gets me very close to matching the packets sent by the app:
var formData = new FormUrlEncodedContent(new[]
{
new KeyValuePair<string, string>(JsonConvert.SerializeObject(myPayload), "")
});
var response = Client.PostAsync(myURL, formData).Result;
var json = response.Content.ReadAsStringAsync().Result;
This gets me almost exactly the same payload sent by the application, except it encodes the data (I know, "encoded" is right there in the name). I need to get the exact same request but without the data being encoded, but I can't quite find the right object(s) to pull it off. How do I keep this payload from being URL encoded?
Edit:
This is a login request I pulled from Wireshark emanating from the application:
POST /Login HTTP/1.1
Content-Type: application/x-www-form-urlencoded
Host: 1.1.1.1
Content-Length: 161
Expect: 100-continue
Connection: Close
{"username":"myuser","auth-id":"0a0a140f81a2ce0c303386e93cec41bf04660c22a881be9a"}
This is what the above will generate:
POST /Login HTTP/1.1
Expect: 100-continue
Connection: Close
Content-Type: application/x-www-form-urlencoded
Content-Length: 221
Host: 1.1.1.1
%7B%22user-name%22%3A%22myuser%22%2C%22auth-id%22%3A%220a0a140f81a2ce0c303386e93cec41bf04660c22a881be9a%22%7D=
I've edited them for brevity so the Content-Length is wrong. I realize it might not be the best way to send this data, but I have no control over how it's consumed.
Since you're actually trying to send JSON, I think you need to wrap the JSON in a StringContent object rather than a FormUrlEncoded object. Form-encoded data and JSON data are two different ways of formatting a payload (another commonly used format would be XML, for example). Using them both together doesn't make any sense.
I think something like this should work:
var content = new StringContent(JsonConvert.SerializeObject(myPayload), Encoding.UTF8, "application/json");
var response = Client.PostAsync(myURL, content).Result;
var json = response.Content.ReadAsStringAsync().Result;
(P.S. the Content-Type: application/x-www-form-urlencoded header sent by the application appears to be misleading, since the request body clearly contains JSON. Presumably the receiving server is tolerant of this nonsense, or just ignores it because it's always expecting JSON.)
I looked at the MS Source code according to their interpretation the HttpClient itself does not have "Content-Type" only the content should have content-type. Seems logical except when you're dealing with MultipartFormDataContent.
MultipartFormDataContent completely ignores the following code:
string boundary = "--" + GenerateRandomString();
using (var content = new MultipartFormDataContent(boundary))
{
content.Headers.ContentType = new MediaTypeHeaderValue($"multipart/form-data");
content.Headers.ContentType.Parameters.Add(new NameValueHeaderValue("boundry", boundary));
...
}
No "Content-Type" is present in the request.
And also ignores:
string boundary = "--" + GenerateRandomString();
using (var content = new MultipartFormDataContent(boundary))
{
content.Headers.Remove("Content-Type");
content.Headers.TryAddWithoutValidation("Content-Type", "multipart/form-data; boundary=" + boundary);
...
}
Attemting to set it at on the HttpClient
client.DefaultRequestHeaders.TryAddWithoutValidation("Content-Type", "multipart/form-data; boundary=" + boundary);
throws the following error:
System.InvalidOperationException: 'Misused header name. Make sure request headers are used with HttpRequestMessage, response headers with HttpResponseMessage, and content headers with HttpContent objects.'
I can find plenty of examples of how to do this using StringContent but none with MultipartFormDataContent. MultipartFormDataContent allows setting the Content-Type and Content-Disposition with each field, I need this more at the client level. I need a header that looks something like this:
accept: application/json
Content-Type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW
Authorization: Basic ZXlKbGRDSTZJakUxTWpZNU9UTXpOTnpkMjl5WkNJNklqa3haRGxtWkdKa1lUazRaVEJqWmpsalpUaGxNV1V3TXpOalxuWmpCbE1tVXhJaXdpZFhObGNpSTZJbUZrYldsdUluMD1cbjo=
Cache-Control: no-cache
Many non-Microsoft APIs require the "boundary" tag so it can distinguish the individual fields of data being sent. The validation here on the request seems a little over the top. Even TryAddWithoutValidation doesn't work (maybe a bug?). I realize that it may be possible to interpret RFC7578 in a way that says it shouldn't be required but flat out not allowing it doesn't seem right to me either. Anyone else ever run into this issue and solve it.
Initially I thought this was an HttpClient bug. I added logging to capture the request and the response. That logging was missing headers which lead me to believe that the missing "multi-part/form-data" content header was the issue and the reason the API I'm using kept telling me it couldn't find a required field. It turns out to be an issue with how the API handles the data it's sent when it's multi-part/form-data. After comparing both my HttpWebRequest and the HttpClient request in fiddler I discovered the following difference in the data being sent:
HttpWebRequest
----vekhftkcthxr
Content-Disposition: form-data; name="name";
d30-20180524
HttpClient
----bcgifxyjkmkw
Content-Type: text/plain; charset=utf-8
Content-Disposition: form-data; name=name
d30-20180524
I build the HttpWebRequest manually so I included the quotes and ending semi-colons. The HttpClient request is built for me and does not include the extra quotes and semi-colon. So the API I am using does not play well with request being generated by the HttpClient even though the request is technically correct.
Thanks to Panagiotis Kanavos for showing me my error.
I am a bit puzzled at the moment. I have a web application that manipulates a file and then returns the file to a user's browser for download when it's done.
The download part is going well, as I'm using Response.AddHeader and Reponse.BinaryWrite to push the file back to the browser but I am unable to call any further methods after using Response methods.
I suppose I have not worked with HttpReponse enough to know the trick to this. Perhaps I would be better off using another class or generic handler to handle the download?
My code goes something like...
// Methods to be called first
Response.Clear();
Response.ClearContent();
Response.ClearHeaders();
Response.AddHeader("Content-Disposition", string.Format("attachment;filename={0}.pdf", "New_Merged_PDF_" + DateTime.Now.ToString("MMMM-dd-yyyy")));
Response.ContentType = "application/pdf";
Response.BinaryWrite(output.ToArray());
Response.End();
// Methods to be called last (these wont work)
Probably something simple that I'm overlooking but I'm still trying to figure it out.
To add a little color to Servy's explaination; there is an order of operations within the HTTP protocol specification. One of them is that the Headers need to be sent to the client before the Body. This allows the receiver of the response to properly deal with the Body that is sent based on any Headers.
The presence of a message-body in a request is signaled by the
inclusion of a Content-Length or Transfer-Encoding header field in
the request's message-headers.
IETF RFC 2616 - Hypertext Transfer Protocol -- HTTP/1.1
One of the few (if the only, I'm not sure) exception is the Content-Type: multipart/mixed;
IETF RFC 1867 - Form-based File Upload in HTML (although Obsolete, examples are still relevant)
Content-type: multipart/form-data, boundary=AaB03x
--AaB03x
content-disposition: form-data; name="field1"
Joe Blow
--AaB03x
content-disposition: form-data; name="pics"; filename="file1.txt"
Content-Type: text/plain
... contents of file1.txt ...
--AaB03x--
I am using the ASP.NET Web API Client Libraries for .NET 4.0 (Microsoft.AspNet.WebApi.Client version 4.0.30506.0).
I need to send an HTTP DELETE with a request body. I have coded it as follows:
using (var client = new HttpClient())
{
client.BaseAddress = Uri;
client.DefaultRequestHeaders.Accept.Clear();
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
// I would normally use httpClient.DeleteAsync but I can't because I need to set content on the request.
// For this reason I use httpClient.SendAsync where I can both specify the HTTP DELETE with a request body.
var request = new HttpRequestMessage(HttpMethod.Delete, string.Format("myresource/{0}", sessionId))
{
var data = new Dictionary<string, object> {{"some-key", "some-value"}};
Content = new ObjectContent<IDictionary<string, object>>(data, new JsonMediaTypeFormatter())
};
var response = await client.SendAsync(request);
// code elided
}
Per Fiddler, the request body is never serialized:
DELETE http://localhost:8888/myApp/sessions/blabla123 HTTP/1.1
Accept: application/json
Content-Type: application/json; charset=utf-8
Host: localhost:8888
Content-Length: 38
Expect: 100-continue
The response from the server:
HTTP/1.1 408 Request body incomplete
Date: Sun, 10 Aug 2014 17:55:17 GMT
Content-Type: text/html; charset=UTF-8
Connection: close
Cache-Control: no-cache, must-revalidate
Timestamp: 13:55:17.256
The request body did not contain the specified number of bytes. Got 0, expected 38
I have tried a number of workarounds, including changing the type being serialized to something else, doing the serialization myself with JsonSerialize, changing the HTTP DELETE to PUT, etc...
Nothing worked. Any help would be much appreciated.
I resolved the issue, though it does not make sense. I noticed that if I changed my call to HTTP PUT or POST, it still failed to serialize the Content as a request body. That was rather strange as previous PUTs and POSTs were successful. After doing a ton of debugging into framework libraries (using Reflector), I finally got to the only thing left that was "different."
I am using NUnit 2.6.2. The structure of my test is:
[Test]
async public void Test()
{
// successful HTTP POST and PUT calls here
// successful HTTP DELETE with request body here (after
// moving it from the TearDown below)
}
[TearDown]
async public void TerminateSession()
{
// failed HTTP DELETE with request body here
}
Why does this fail in the TearDown but not in the Test itself? I have no idea. Is something going on with the TearDown attribute or with the use of the async keyword (since I await async calls)?
I am not sure what it is causing this behavior, but I do know now that I can submit an HTTP DELETE with a request body (as outlined in my code sample in the question).
Another solution that worked is as follows:
[Test]
async public void Test()
{
// create and use an HttpClient here, doing POSTs, PUTs, and GETs
}
// Notice the removal of the async keyword since now using Wait() in method body
[TearDown]
public void TerminateSession()
{
// create and use an HttpClient here and use Wait().
httpClient.SendAsync(httpRequestMessage).Wait();
}
I know it's never quite that helpful to say, "don't do it that way", but in this case I think it makes sense to split the calls into a DELETE followed or preceeded by a POST or PUT.
The HTTP RFC doesn't explicitly opine on the matter, so technically it means that we can. The other question, however, is should we do it.
In cases such as this I would look for other implementations to see what is the de facto standard. As you've found in the .net implementation, it appears that the designers did not expect to send a body with the DELETE call. So, let's look at another popular (and very different impl) Python Requests:
>>> r = requests.delete(url=url, auth=auth)
>>> r.status_code
204
>>> r.headers['status']
'204 No Content'
No body here other. So, if the spec authors didn't mention it, and popular implementations assume that there's no body, then the principle of least surprise means we shouldn't do it either.
So, if you can change the API, it will be easier on clients of the API to split into two calls. Otherwise, you'll likely have to resort to custom hackery to cram the body into a DELETE call.
The good news is that you've likely found a bug in the .net framework, which is an achievement in and of itself. Clients advertising a non-zero Content-Length without actually sending it are broken.
In case anybody else runs into this, one thing I've noticed that can cause this is if you set a header with a newline in it.
We had an encrypted OAuth token, which gets decrypted at runtime and set as the OAuth header on the app. The newline was encrypted into the token, so it was not obvious from looking at the configs or anything that it was there, but if you do:
var message = new HttpRequestMessage(HttpMethod.Post, "https://example.com");
message.Headers.ContentType = new MediaTypeHeaderValue("application/json");
message.Content = new StringContent("{ \"someKey\": \"someValue\" }", Encoding.UTF8);
// note the trailing newline
message.Headers.Authorization = new AuthenticationHeaderValue("OAuth", "my auth token\n");
var response = await httpClient.SendAsync(request);
The HTTP request will be sent, but the content will not be sent with it. There are no exceptions thrown when this happens and if you inspect the HttpRequestMessage, the content will appear to be there, but it does not actually get sent over the wire.
This happens in .NET 5 on Windows and Linux, I haven't tested it on other framework versions/platforms.
I am attempting to upload a csv file but I can't seem to get it to work programatically. If I use Postman in Chrome to send the file it works and here is what it sends (Fiddler output):
------WebKitFormBoundary2YsMyLR3QAPruTy4
Content-Disposition: form-data; name="Content-Type"; filename="613022.csv"
Content-Type: application/vnd.ms-excel
// File Content here
------WebKitFormBoundary2YsMyLR3QAPruTy4--
However, using this code:
WebClient wc = new WebClient();
wc.Credentials = new NetworkCredential(user, pw);
wc.Headers[HttpRequestHeader.Cookie] = "OBBasicAuth=fromDialog";
wc.Headers[HttpRequestHeader.ContentType] = "application/vnd.ms-excel";
wc.UploadFile(baseURL + service + apiVersion + resource, "post", file);
Results in (Fiddler output):
-----------------------8d101dbe85fe96c
Content-Disposition: form-data; name="file"; filename="613022.csv"
Content-Type: application/vnd.ms-excel
// File Content here
-----------------------8d101dbe85fe96c--
Which does not work and the server returns a 503 error. The only difference I see is in the Content-Disposition name. How can I set this or is there a better way to accomplish this?
Perhaps you need "upload" instead of "post" in the method parameters - just guessing.
I believe the problem with your original approach is that WebClient.UploadFile generates the multipart/form-data request in a way that is unexpected by the server, hence the 5xx error code.
After looking around for a bit, I think the answer to this question should give you a starting point to tweak the request according to your needs.