I have tried unseccessfully to send the next header to the Twitter API:
POST /oauth2/token HTTP/1.1
Host: api.twitter.com
User-Agent: My Twitter App v1.0.23
Authorization: Basic eHZ6MWV2R ... o4OERSZHlPZw==
Content-Type: application/x-www-form-urlencoded;charset=UTF-8
Content-Length: 29
Accept-Encoding: gzip
grant_type=client_credentials
I searched and found that there's a method called WebClient.UploadData(this method, implicitly sets HTTP POST as the request method) but I dont really
know how to work with it.
I know how to change the current headers using Set method.
But what about the HTTP body message? how can I add some body to the header?(grant_type)
PS: I read the documention.
Not much to it unless you are dealing with multi-part data. Just build a string with the post data (URL encode if the data requires it), get the bytes, set the content-length, and write your data to the request stream.
string postData = String.Format("field1=value1&field2=value2");
byte[] postBytes = System.Text.ASCIIEncoding.UTF8.GetBytes(postData);
HttpWebRequest req = HttpWebRequest.Create("http://myurl.com");
// ... other request setup stuff
req.ContentLength = postBytes.Length;
using (var stream = req.GetRequestStream())
{
stream.Write(postBytes, 0, postBytes.Length);
}
Related
I'm trying to "repurpose" a third-party API used by a desktop application. I've found that the below code gets me very close to matching the packets sent by the app:
var formData = new FormUrlEncodedContent(new[]
{
new KeyValuePair<string, string>(JsonConvert.SerializeObject(myPayload), "")
});
var response = Client.PostAsync(myURL, formData).Result;
var json = response.Content.ReadAsStringAsync().Result;
This gets me almost exactly the same payload sent by the application, except it encodes the data (I know, "encoded" is right there in the name). I need to get the exact same request but without the data being encoded, but I can't quite find the right object(s) to pull it off. How do I keep this payload from being URL encoded?
Edit:
This is a login request I pulled from Wireshark emanating from the application:
POST /Login HTTP/1.1
Content-Type: application/x-www-form-urlencoded
Host: 1.1.1.1
Content-Length: 161
Expect: 100-continue
Connection: Close
{"username":"myuser","auth-id":"0a0a140f81a2ce0c303386e93cec41bf04660c22a881be9a"}
This is what the above will generate:
POST /Login HTTP/1.1
Expect: 100-continue
Connection: Close
Content-Type: application/x-www-form-urlencoded
Content-Length: 221
Host: 1.1.1.1
%7B%22user-name%22%3A%22myuser%22%2C%22auth-id%22%3A%220a0a140f81a2ce0c303386e93cec41bf04660c22a881be9a%22%7D=
I've edited them for brevity so the Content-Length is wrong. I realize it might not be the best way to send this data, but I have no control over how it's consumed.
Since you're actually trying to send JSON, I think you need to wrap the JSON in a StringContent object rather than a FormUrlEncoded object. Form-encoded data and JSON data are two different ways of formatting a payload (another commonly used format would be XML, for example). Using them both together doesn't make any sense.
I think something like this should work:
var content = new StringContent(JsonConvert.SerializeObject(myPayload), Encoding.UTF8, "application/json");
var response = Client.PostAsync(myURL, content).Result;
var json = response.Content.ReadAsStringAsync().Result;
(P.S. the Content-Type: application/x-www-form-urlencoded header sent by the application appears to be misleading, since the request body clearly contains JSON. Presumably the receiving server is tolerant of this nonsense, or just ignores it because it's always expecting JSON.)
I looked at the MS Source code according to their interpretation the HttpClient itself does not have "Content-Type" only the content should have content-type. Seems logical except when you're dealing with MultipartFormDataContent.
MultipartFormDataContent completely ignores the following code:
string boundary = "--" + GenerateRandomString();
using (var content = new MultipartFormDataContent(boundary))
{
content.Headers.ContentType = new MediaTypeHeaderValue($"multipart/form-data");
content.Headers.ContentType.Parameters.Add(new NameValueHeaderValue("boundry", boundary));
...
}
No "Content-Type" is present in the request.
And also ignores:
string boundary = "--" + GenerateRandomString();
using (var content = new MultipartFormDataContent(boundary))
{
content.Headers.Remove("Content-Type");
content.Headers.TryAddWithoutValidation("Content-Type", "multipart/form-data; boundary=" + boundary);
...
}
Attemting to set it at on the HttpClient
client.DefaultRequestHeaders.TryAddWithoutValidation("Content-Type", "multipart/form-data; boundary=" + boundary);
throws the following error:
System.InvalidOperationException: 'Misused header name. Make sure request headers are used with HttpRequestMessage, response headers with HttpResponseMessage, and content headers with HttpContent objects.'
I can find plenty of examples of how to do this using StringContent but none with MultipartFormDataContent. MultipartFormDataContent allows setting the Content-Type and Content-Disposition with each field, I need this more at the client level. I need a header that looks something like this:
accept: application/json
Content-Type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW
Authorization: Basic ZXlKbGRDSTZJakUxTWpZNU9UTXpOTnpkMjl5WkNJNklqa3haRGxtWkdKa1lUazRaVEJqWmpsalpUaGxNV1V3TXpOalxuWmpCbE1tVXhJaXdpZFhObGNpSTZJbUZrYldsdUluMD1cbjo=
Cache-Control: no-cache
Many non-Microsoft APIs require the "boundary" tag so it can distinguish the individual fields of data being sent. The validation here on the request seems a little over the top. Even TryAddWithoutValidation doesn't work (maybe a bug?). I realize that it may be possible to interpret RFC7578 in a way that says it shouldn't be required but flat out not allowing it doesn't seem right to me either. Anyone else ever run into this issue and solve it.
Initially I thought this was an HttpClient bug. I added logging to capture the request and the response. That logging was missing headers which lead me to believe that the missing "multi-part/form-data" content header was the issue and the reason the API I'm using kept telling me it couldn't find a required field. It turns out to be an issue with how the API handles the data it's sent when it's multi-part/form-data. After comparing both my HttpWebRequest and the HttpClient request in fiddler I discovered the following difference in the data being sent:
HttpWebRequest
----vekhftkcthxr
Content-Disposition: form-data; name="name";
d30-20180524
HttpClient
----bcgifxyjkmkw
Content-Type: text/plain; charset=utf-8
Content-Disposition: form-data; name=name
d30-20180524
I build the HttpWebRequest manually so I included the quotes and ending semi-colons. The HttpClient request is built for me and does not include the extra quotes and semi-colon. So the API I am using does not play well with request being generated by the HttpClient even though the request is technically correct.
Thanks to Panagiotis Kanavos for showing me my error.
I am trying to call a REST service from a C# ASP.NET 4.0 application using RestSharp.
It's a fairly straightforward POST call to a https:// address; my code is something like this (CheckStatusRequest is a plain simple DTO with about four or five string and int properties - nothing fancy):
public CheckStatusResponse CheckStatus(CheckStatusRequest request) {
// set up RestClient
RestClient client = new RestClient();
string uri = "https://.......";
// create the request (see below)
IRestRequest restRequest = CreateRequestWithHeaders(url, Method.POST);
// add the body to the request
restRequest.AddBody(request);
// execute call
var restResponse = _restClient.Execute<CheckStatusResponse>(restRequest);
}
// set up request
private IRestRequest CreateRequestWithHeaders(string uri, Method method) {
// define request
RestRequest request = new RestRequest(uri, method);
// add two required HTTP headers
request.AddHeader("Accept", "application/json");
request.AddHeader("Content-Type", "application/json");
// define JSON as my format
request.RequestFormat = DataFormat.Json;
// attach the JSON.NET serializer for RestSharp
request.JsonSerializer = new RestSharpJsonNetSerializer();
return request;
}
The problem I'm having when I send these requests through Fiddler to see what's going on is that my request suddenly gets a third and unwanted HTTP header:
POST https://-some-url- HTTP/1.1
Accept: application/json
User-Agent: RestSharp/104.4.0.0
Content-Type: application/json
Host: **********.com
Content-Length: 226
Accept-Encoding: gzip, deflate <<<=== This one here is UNWANTED!
Connection: Keep-Alive
I suddenly have that Accept-Encoding HTTP header, which I never specified (and which I don't want to have in there). And now my response is no longer proper JSON (which I'm able to parse), but suddenly I get back gzipped binary data instead (which doesn't do real well when trying to JSON-deserialize)....
How can I get rid of that third unwanted HTTP header?
I tried to set it to something else - whatever I enter just gets appended to those settings
I tried to somehow "clear" that HTTP header - without any success
I tried finding a property on the RestClient or the RestRequest classes to specify "do not use GZip"
Looking at the sources (Http.Sync.cs and Http.Async.cs) of RestSharp you can see that these values are hardcoded:
webRequest.AutomaticDecompression =
DecompressionMethods.Deflate | DecompressionMethods.GZip | DecompressionMethods.None;
There is also an open issue that describes this problem. It was opened August 2014 but still not solved. I think you can leave a comment there and maybe they will pay attention.
When it comes to web development, I know very very little...
I have found some code and explanations from the following site. https://dev.twitter.com/docs/auth/implementing-sign-twitter
Ultimately, I want to implement login with twitter. But I am having trouble rewriting those POST web requests into a c# HttpWebRequest format that I can reuse in the rest of our apps. If we examine the first webrequest made...
POST /oauth/request_token HTTP/1.1
User-Agent: themattharris' HTTP Client
Host: api.twitter.com
Accept: */*
Authorization:
OAuth oauth_callback="http%3A%2F%2Flocalhost%2Fsign-in-with-twitter%2F",
oauth_consumer_key="cChZNFj6T5R0TigYB9yd1w",
oauth_nonce="ea9ec8429b68d6b77cd5600adbbb0456",
oauth_signature="F1Li3tvehgcraF8DMJ7OyxO4w9Y%3D",
oauth_signature_method="HMAC-SHA1",
oauth_timestamp="1318467427",
oauth_version="1.0"
I want to transform that into a working HttpWebRequest.
Thus far. My code looks like this...
HttpWebRequest httpReq = (HttpWebRequest)WebRequest.Create("https://api.twitter.com/oauth/request_token");
ASCIIEncoding encoding = new ASCIIEncoding();
httpReq.Method = "POST";
httpReq.ContentType = "application/x-www-form-urlencoded";
httpReq.Accept = "Accept=text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
This is unfortunately how far I did get... I don't know how these requests work. I need to include the rest of the data and make the call. But I am stuck. Any help would be greatly appreciated.
Try this :
ASCIIEncoding encoder = new ASCIIEncoding();
byte[] data = encoder.GetBytes(serializedObject); // the data you wanted to send
HttpWebRequest request = new WebRequest.Create("https://api.twitter.com/oauth/request_token") as HttpWebRequest;
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded";
request.ContentLength = data.Length;
request.GetRequestCode().Write(data, 0, data.Length);
Also a possible dublicate (similar question) : Why I get 411 Length required error?
So I'm in the process of attempting to simulate multiple logins all generating exceptions at the same time on our corporate website for the purpose of testing our logging framework (which we think there may be issues with thread synchronization). Anyway, so I need to log in to our website programatically. Here's what I have so far:
// Block 1
Uri url = new Uri("http://withheld");
HttpWebRequest request = WebRequest.Create(url) as HttpWebRequest;
request.Method = "GET";
HttpWebResponse response = request.GetResponse() as HttpWebResponse;
string viewState = string.Empty;
string previousPage = string.Empty;
string eventValidation = string.Empty;
using (StreamReader reader = new StreamReader(response.GetResponseStream()))
{
string strResponse = reader.ReadToEnd();
viewState = HttpUtility.UrlEncode(GetTagValue(strResponse, "__VIEWSTATE"));
previousPage = HttpUtility.UrlEncode(GetTagValue(strResponse, "__PREVIOUSPAGE"));
eventValidation = HttpUtility.UrlEncode(GetTagValue(strResponse, "__EVENTVALIDATION"));
}
// Block 2
string username = "user01";
string password = "password99";
HttpWebRequest request2 = WebRequest.Create(url) as HttpWebRequest;
request2.KeepAlive = true;
request2.Method = "POST";
request2.ContentType = "application/x-www-form-urlencoded";
string postData = string.Format("__LASTFOCUS=&ctrlCreateNewPassword_scriptManagerMaster_HiddenField=&__EVENTTARGET=&__EVENTARGUMENT=&__VIEWSTATE={0}&__PREVIOUSPAGE={1}&__EVENTVALIDATION={2}&UserName={3}&Password={4}&LoginButton=Log+in", new string[] { viewState, previousPage, eventValidation, username, password});
byte[] dataBytes = UTF8Encoding.UTF8.GetBytes(postData);
request2.ContentLength = dataBytes.Length;
using (Stream postStream = request2.GetRequestStream())
{
// Here's the problem
postStream.Write(dataBytes, 0, dataBytes.Length);
}
HttpWebResponse httpResponse = request2.GetResponse() as HttpWebResponse;
// At this point httpResponse.Cookies is null
// I believe it's because the line above has actually initiated another
// request/response which DOES NOT include the authentication cookie.
// See fiddler output below to understand why I think that.
// Block 3
//Uri url2 = new Uri("http://Withheld/GenerateException.aspx");
//http = WebRequest.Create(url2) as HttpWebRequest;
//http.CookieContainer = new CookieContainer();
//http.CookieContainer.Add(httpResponse.Cookies);
//HttpWebResponse httpResponse2 = http.GetResponse() as HttpWebResponse;
Looks reasonably straightforward right? Well it doesn't work. I don't know whether I need the viewState and whatnot or not, but I figured I'd mimic what a regular browser does as closely as possible.
Near as I can tell what's happening is this:
We hit the page with a simple GET. That gives us the viewstate, etc which is parsed out to be included in the following request.
We now post the viewstate, username, password, etc to the server using postStream.Write(). The server at this point responds with an authentication cookie, and forwards us off to /Default.aspx.
Now we execute reqest2.GetResponse(), but instead of getting the response that forwarded us to /default.aspx and had he authentication cookie, it looks like this line is actually causing ANOTHER request to get us the resource at /default.aspx. The problem is httpWebResponse DOES NOT include the authentication cookie we need for the next request (which is commented out currently).
Why? Why is this behaving in this manor, and how do I handle it properly. Here's the output from fiddler to further explain what's going on:
Block 1 generates this request/response
Request Header:
GET http://withheld/Login.aspx HTTP/1.1
Host: withheld
Connection: Keep-Alive
Response Header:
HTTP/1.1 200 OK
Connection: close
Date: Mon, 04 Feb 2013 16:37:37 GMT
Server: Microsoft-IIS/6.0
X-Powered-By: ASP.NET
Set-Cookie: .EXTASPXAUTH=; expires=Tue, 12-Oct-1999 04:00:00 GMT; path=/; HttpOnly
Cache-Control: private, no-cache="Set-Cookie"
Content-Type: text/html; charset=utf-8
Content-Length: 16975
Response is the actual login webpage. Omitted for obvious reasons.
Stepping through the code, this request/response is generated immediately following the call to postStream.Write:
Request:
POST http://Withheld/Login.aspx HTTP/1.1
Content-Type: application/x-www-form-urlencoded
Host: withheld
Content-Length: 2109
Expect: 100-continue
__LASTFOCUS=&ctrlCreateNewPassword_scriptManagerMaster_HiddenField=&__EVENTTARGET=&__EVENTARGUMENT=&__VIEWSTATE=%2fwEPDwUJNTE1NTIxNjkxD2QWAgIDD2QWBgIDDxYCHgVjbGFzcwUQaGVhZGVyX2Jhbm5lcl9lbhYCAgEPFgQeB29uY2xpY2sFI3dpbmRvdy5sb2NhdGlvbj0naHR0cDovL3d3dy5tZG0uY2EnHwAFFmhlYWRlcl9iYW5uZXJfZW5fc21hbGxkAgUPZBYQAgcPDxYEHhdQYXJ0aWFsUmVuZGVyaW5nQ2hlY2tlZGceGUlzUGFydGlhbFJlbmRlcmluZ0VuYWJsZWRoZGQCCQ8PFgQfAmcfA2hkZAIPDw8WBB8CZx8DaGRkAhEPDxYCHg1PbkNsaWVudENsaWNrBcEBamF2YXNjcmlwdDpQYWdlX0NsaWVudFZhbGlkYXRlKCdMb2dpbkN0cmwnKTsgaWYgKFBhZ2VfSXNWYWxpZCkgeyBMb2dpbkJ1dHRvbi5zdHlsZS52aXNpYmlsaXR5ID0gJ2hpZGRlbic7IExvZ2luQnV0dG9uLnN0eWxlLmRpc3BsYXkgPSAnbm9uZSc7IExvZ2luQnV0dG9uRGlzYWJsZWQuc3R5bGUudmlzaWJpbGl0eSA9ICd2aXNpYmxlJzsgfWRkAhkPDxYCHgRUZXh0Bc0BSWYgeW91IHJlcXVpcmUgYXNzaXN0YW5jZSwgb3VyIE1EIE9ubGluZSBTdXBwb3J0IFNwZWNpYWxpc3RzIGNhbiBiZSByZWFjaGVkIGF0ICg4NzcpIDQzMS0wMzMwIG9yIGJ5IGUtbWFpbCBhdCA8YSBocmVmPSJtYWlsdG86d2Vic3VwcG9ydEBjbWEuY2EiPndlYnN1cHBvcnRAY21hLmNhPC9hPi48YnI%2bPGJyPldlIGVuY291cmFnZSB5b3UgdG8gcmV2aWV3IHRoZWRkAhsPDxYCHwQFOXdpbmRvdy5vcGVuKCdodHRwOi8vMTkyLjE2OC4xNjUuMzIvbGVnYWwvJyk7cmV0dXJuIGZhbHNlO2RkAh8PDxYCHwQFRndpbmRvdy5vcGVuKCdodHRwOi8vMTkyLjE2OC4xNjUuMzIvc3lzdGVtLWVuaGFuY2VtZW50LycpO3JldHVybiBmYWxzZTtkZAIhDw8WAh8EBUZ3aW5kb3cub3BlbignaHR0cDovLzE5Mi4xNjguMTY1LjMyL3N5c3RlbS1lbmhhbmNlbWVudC8nKTtyZXR1cm4gZmFsc2U7ZGQCCQ9kFgICAQ9kFgICAg9kFgJmD2QWAgIBD2QWAgIdDw8WAh8EBfsBamF2YXNjcmlwdDpjdHJsQ3JlYXRlTmV3UGFzc3dvcmRfQ3JlYXRlTmV3UGFzc3dvcmRQdXNoQnV0dG9uLnN0eWxlLnZpc2liaWxpdHkgPSAnaGlkZGVuJzsgY3RybENyZWF0ZU5ld1Bhc3N3b3JkX0NyZWF0ZU5ld1Bhc3N3b3JkUHVzaEJ1dHRvbi5zdHlsZS5kaXNwbGF5ID0gJ25vbmUnOyBjdHJsQ3JlYXRlTmV3UGFzc3dvcmRfQ3JlYXRlTmV3UGFzc3dvcmRQdXNoQnV0dG9uRGlzYWJsZWQuc3R5bGUudmlzaWJpbGl0eSA9ICd2aXNpYmxlJztkZBgBBR5fX0NvbnRyb2xzUmVxdWlyZVBvc3RCYWNrS2V5X18WAQUMaWJ0bk1vcmVJbmZvdZmFXkMbPfVWPQYtreXvFt8Bck8%3d&__PREVIOUSPAGE=1aYW5DqTKrT4ieGPkHcnrQLIq8lEcSIVkql1EugwSQNV_5102t5D7QDmOnuQFA4Tz9Mh5-CEYpkRngMROFFeeAG12Ss1&__EVENTVALIDATION=%2fwEWCQKKr%2bXcBgKvruq2CALSxeCRDwL%2bjNCfDwKH8YSKBgKN6O7XCwKz9P38DALl3I74DwLWxI74D6Nz%2f2bCBFC%2bM9glZmEyM%2byOCTZg&UserName=user01&Password=password99&LoginButton=Log+in
Response:
HTTP/1.1 302 Found
Connection: close
Date: Mon, 04 Feb 2013 16:36:55 GMT
Server: Microsoft-IIS/6.0
X-Powered-By: ASP.NET
Location: /Default.aspx?locale=en
Set-Cookie: .EXTASPXAUTH=65BB5BFDD274F730E26CAEAAEB417792A764E7B8E8C6C9AC8C47FA97EF35DFACF551A53EAA6EA67D868C8A9BF55EBA758A5E724C58269028EE48F56268A204CBED19B60FC1AF58892989D9546202C037E97BF0EEE6A6281FF5EEA461BC30C5C7A71DFD64027AEB796D3FD21AE97ECFB16FF0F95C; path=/; HttpOnly
Cache-Control: private, no-cache="Set-Cookie"
Content-Type: text/html; charset=utf-8
Content-Length: 140
<html><head><title>Object moved</title></head><body>
<h2>Object moved to here.</h2>
</body></html>
Note that the above response includes an authenication cookie.
Now we run the following line of code with the intention of getting that cookie:
HttpWebResponse httpResponse = request2.GetResponse() as HttpWebResponse;
But instead the following request/response is generated in fiddler:
Request:
GET http://withtheld/Default.aspx?locale=en HTTP/1.1
Content-Type: application/x-www-form-urlencoded
Host: withheld
Response:
HTTP/1.1 302 Found
Connection: close
Date: Mon, 04 Feb 2013 16:37:38 GMT
Server: Microsoft-IIS/6.0
X-Powered-By: ASP.NET
Location: /Login.aspx?ReturnUrl=%2fDefault.aspx%3flocale%3den&locale=en
Cache-Control: private
Content-Type: text/html; charset=utf-8
Content-Length: 182
<html><head><title>Object moved</title></head><body>
<h2>Object moved to here.</h2>
</body></html>
I believe this is the response that httpResponse now contains. How can I actually get the cookie to request up another protected page after the login is done?
Thanks!
Well it turns out I had two problems here. One is that I needed to call this:
request.AllowAutoRedirect = false
This keeps the framework from just skipping the response that had the authentication cookie in it, and actually gives back the response we're interested in.
The other issue was that you have to create a new instance of a CookieContainer and assign it to the request. Without having done that Response.Cookies contains no cookies. As soon as you've assigned your own container it's populated after the response is made. I don't know why.
When you login, the authentication token will be sent as a cookie from the server to your client. You will need to store this cookie, and then re-send it with every future request. Re-sending the cookie tells the server that you have been authenticated as a certain user.
To get the cookies that were sent in response after you logged in:
HttpCookieCollection loginResponseCookies = Response.Cookies;
This collection will include the auth cookie, any session cookies, etc.
Then, just re-send these cookies with every subsequent request, and the server will authenticate you as a result.
foreach(HttpCookie loginResponseCookie in loginResponseCookies)
{
Response.Cookies.Add(loginResponseCookie);
}