So I'm attempting to post a form using a HttpClient (and CloudFlareUtilities), and initialise it as shown below:
var handler = new ClearanceHandler {
MaxRetries = 4,
};
var client = new HttpClient(handler);
client.DefaultRequestHeaders.Connection.Clear();
client.DefaultRequestHeaders.ConnectionClose = false;
client.DefaultRequestHeaders.Add("User-Agent",
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 " +
"(KHTML, like Gecko) Chrome/64.0.3282.186 Safari/537.36");
client.Timeout = TimeSpan.FromSeconds(30);
Creating the client is successful, with no apparent issues occurring. However, I create a POST request like below:
var content = new FormUrlEncodedContent(new[] {
new KeyValuePair<string, string>("login", "username"),
new KeyValuePair<string, string>("password", "pw"),
new KeyValuePair<string, string>("cookie_check", "1"),
new KeyValuePair<string, string>("redirect", "myRedirect"),
new KeyValuePair<string, string>("register", "0"),
new KeyValuePair<string, string>("remember", "1")
});
and post the data:
var post = client.PostAsync("https://www.mc-market.org/login/login", content).Result.Content.ReadAsStreamAsync().Result;
The aforementioned line throws the error(s):
Inner Exception 1:
HttpRequestException: Error while copying content to a stream.
Inner Exception 2:
IOException: The read operation failed, see inner exception.
Inner Exception 3:
WinHttpException: The operation has been canceled
What I can interpret from this is that something went wrong, and something cancelled the task, yet I never cancel it myself.
Limiting errored line to
var post = client.PostAsync("https://www.mc-market.org/login/login", content);
gives the same error.
Is it possible I've created the Content incorrectly? Some indication of where I may have gone is appreciated.
I decided not to go with the HttpClient, for simplicity.
Instead I opted to go with PhantomJS & Selenium, which natively supports Javascript in-browser anyway (which I decided to enable for the Cloudflare bypass).
What I then did is used its SendKeys() method to type in the login fields for in the website https://www.mc-market.org/login/ (instead of https://www.mc-market.org/login/login), and press the button with Click(). An example is shown below;
driver.FindElementById("ctrl_pageLogin_login").SendKeys("NAME");
driver.FindElementById("ctrl_pageLogin_password").SendKeys("PASS");
driver.FindElementById("ctrl_pageLogin_remember").Click();
driver.FindElementByCssSelector("input.button.primary").Click();
The result was much simpler and it worked like a charm.
Related
Here is my code:
var cclient = new RestClient("ex.com");
var reqq = new RestRequest("ex", Method.POST);
reqq.AddHeader("Cookie", coockie);
reqq.AddHeader("Host", "profile.callofduty.com");
reqq.AddHeader("Pragma", "no-cache");
reqq.AddHeader("User-Agent", "Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko");
reqq.AddParameter("username", log);
reqq.AddParameter("password", pas);
var responsee = cclient.Post(reqq);
but on http debugger when I inspect the request:
I Don't now the problem please help. none of headers submited
AddHeader(key, value) method is a "proxy" for AddParameter(key, value, ParameterType.HttpHeader) method. All your headers are applied to the request and can be found using ordinary debugger if you inspect Parameters property of your RestRequest instance (in your code sample it's reqq).
Also to send your request you really should use the Execute(IRestRequest) method in your RestClient
I'm trying to imitate a browser from my console application reading from websockets
var tParam = "MoC4gE6";
var sessionId = GetSid(tParam); // simple GET, works the same way as in a browser
HttpGetSomeBytes(sessionId, tParam); // simple GET, return the same bytes as it is in a browser
var uri2 = new Uri($"ws://{_host}/driver.socket/?EIO=3&transport=websocket&sid={sessionId}");
var ws2 = new ClientWebSocket();
Random rand = new Random();
Byte[] randomBytes = new Byte[16];
rand.NextBytes(randomBytes);
ws2.Options.SetRequestHeader("Host", _host);
ws2.Options.SetRequestHeader("Accept-Encoding", "gzip, deflate");
ws2.Options.SetRequestHeader("User-Agent",
"Mozilla/5.0(Windows NT 10.0; Win64; x64) AppleWebKit/537.36(KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36");
await ws2.ConnectAsync(uri2, CancellationToken.None);
ArraySegment<Byte> buffer = new ArraySegment<byte>(new Byte[8192]);
var receiveTask = await ws2.ReceiveAsync(buffer, CancellationToken.None);
The problem is that call ReceiveAsync never returns back.
The server is not mine so I can't check what's going on there.
Update:
I'm able to see 2 responses:
If I send '2' I received '3' as in a browser
If I send '2probe' I received '3probe' as in a browser
Bu when I send '5' expecting to get back some useful data (as I do in a browser) the ReceiveAsync never returns.
I have this REST api (GET) which works fine in browser and spits JSON. I even tried the same API in Excel-VBA using XMLHTTP and it works fine.
But when trying to use the same API in C#, I am getting errors.
First I was getting:
"The underlying connection was closed: An unexpected error occurred on
a send."
on the the line
HttpResponseMessage response = client.GetAsync(urlParameters).Result;
Then I set the security protocol to `Tls11' and that error vanished.
Now I am geeting 404 in the resposne. The URL is correct. I am able to run the same URL in web browser/VBA but not in C#.
Any suggestions or help?
Sorry, can't share the actual url as its work related.
private const string URL = "https://xxx-production-api.abc.com/api/listings/1790956";
private string urlParameters = "?apiToken=64842d73-9761-456b-86fa-a75a409273ce";
public string Download()
{
HttpClient client = new HttpClient();
client.BaseAddress = new Uri(URL);
client.DefaultRequestHeaders.Accept.Add(
new MediaTypeWithQualityHeaderValue("application/json"));
client.DefaultRequestHeaders.Add("Accept-Language", "en-GB,en-US;q=0.8,en;q=0.6,ru;q=0.4");
// List data response.
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls11;
HttpResponseMessage response = client.GetAsync(urlParameters).Result; // Blocking call!
if (response.IsSuccessStatusCode)
Console.WriteLine("Worked");
else
Console.WriteLine("{0} ({1})", (int)response.StatusCode, response.ReasonPhrase);
}
Answering my own question :
Adding the user agent to headers solved the issue for me: client.DefaultRequestHeaders.Add("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36")
Copied and pasted the header info from Chrome dev tool and now the url is working in C# too.
Make sure your work firewall/proxy is not blocking the call.
I have run into a similar issues and this is the final code I use for my GET calls, see if it works for you. The main difference is that I don't use ".Result" I use "GetAwaiter().GetResult()".
using (var client = new HttpClient())
{
var method = string.Format("{0}{1}", ApiUrl, urlPart);
client.BaseAddress = new Uri(ApiUrl);
client.DefaultRequestHeaders.Accept.Clear();
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/xml"));
HttpResponseMessage response = client.GetAsync(method).GetAwaiter().GetResult();
if (response.IsSuccessStatusCode)
{
var data = response.Content.ReadAsStringAsync().GetAwaiter().GetResult();
return data;
}
return default(T);
}
I'm practicing my skills with httpclient .NET 4.5 , but I ran into some trouble trying to get html content from a webpage which I can view no problem with my browser, but I will get a 410 Gone when I use httpclient
There is the first page that I can actually get with httpclient
https://selfservice.mypurdue.purdue.edu/prod/bwckschd.p_disp_detail_sched?term_in=201610&crn_in=20172
But links inside of this above url, like the url represented by "View Catalog Entry "
when I try to access with httpclient, I will get a 410 Gone.
I used Tuple to set up the series of requests
var requests = new List<Tuple<Httpmethod, string, FormUrlEncodedContent, string>>()
{
new Tuple<Httpmethod, string, FormUrlEncodedContent, string>(Httpmethod.GET, "https://selfservice.mypurdue.purdue.edu/prod/bwckschd.p_disp_detail_sched?term_in=201610&crn_in=20172", null, ""),
new Tuple<Httpmethod, string, FormUrlEncodedContent, string>(Httpmethod.GET, "https://selfservice.mypurdue.purdue.edu/prod/bwckctlg.p_display_courses?term_in=201610&one_subj=EPCS&sel_crse_strt=10100&sel_crse_end=10100&sel_subj=&sel_levl=&sel_schd=&sel_coll=&sel_divs=&sel_dept=&sel_attr=", null, ""),
new Tuple<Httpmethod, string, FormUrlEncodedContent, string>(Httpmethod.GET, "https://selfservice.mypurdue.purdue.edu/prod/bwckctlg.p_disp_listcrse?term_in=201610&subj_in=EPCS&crse_in=10100&schd_in=%", null, "")
};
Then use the foreach to loop through pass each element in requests to a function
HttpClientHandler handler = new HttpClientHandler()
{
CookieContainer = cookies,
AllowAutoRedirect = false,
AutomaticDecompression = DecompressionMethods.GZip
};
HttpClient client = new HttpClient(handler as HttpMessageHandler)
{
BaseAddress = new Uri(url),
Timeout = TimeSpan.FromMilliseconds(20000)
};
client.DefaultRequestHeaders.TryAddWithoutValidation("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8");
client.DefaultRequestHeaders.TryAddWithoutValidation("Accept-Encoding", "gzip, deflate, sdch");
client.DefaultRequestHeaders.TryAddWithoutValidation("Accept-Language", "en-US,en;q=0.8,zh-CN;q=0.6,zh;q=0.4");
client.DefaultRequestHeaders.TryAddWithoutValidation("User-Agent", "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36");
client.DefaultRequestHeaders.Connection.Add("keep-alive");
if (referrer.Length > 0)
client.DefaultRequestHeaders.Referrer = new Uri(referrer);
System.Diagnostics.Debug.WriteLine("Navigating to '" + url + "...");
HttpResponseMessage result = null;
//List<Cookie> cook = GetAllCookies(cookies);
try
{
switch (method)
{
case Httpmethod.POST:
result = await client.PostAsync(url, post_content);
break;
case Httpmethod.GET:
result = await client.GetAsync(url);
break;
}
}
catch (HttpRequestException ex)
{
throw new ApplicationException(ex.Message);
}
I have no clue what is wrong with my method, since I have successfully handled requests that needs to be logged in with this method.
I intended to build some sort of web crawler using this.
Please help, thanks in advance.
I forgot to mention, the links inside of the above url could be accessed by clicking on them on a web browser, simply copy them to address bar will get 410 Gone as well.
EDIT: There is some JavaScript code I saw in page source, it's creating an XMLHttpRequest to get the content, so it might be that it only refresh part of the page instead of creating a new webpage. But the url is changed. How do I get httpclient do the clicking thing?
i'm using an HttpClient to create a HTTP request and that client comes from the assembly Windows.Web.Http
All is good when posting the request without the Content-Type Header but the server does not return what I need because it needs that header, so after finding the correct headers needed to be sent I'm facing another problem... I'm not being able to set the Content-Type header
Here is my code (where is the try block is where the error is)
using (var wp = new Windows.Web.Http.HttpClient())
{
HttpRequestMessage mSent = new HttpRequestMessage(HttpMethod.Post, new Uri(url));
//mSent.Headers.Add("Host", "academicos.ubi.pt");
//mSent.Headers.Add("Connection", "keep-alive");
//mSent.Headers.Add("Content-Length", "18532");
//mSent.Headers.Add("Origin", "https://academicos.ubi.pt");
//mSent.Headers.Add("X-Requested-With", "XMLHttpRequest");
//mSent.Headers.Add("Cache-Control", "no-cache");
//mSent.Headers.Add("X-MicrosoftAjax", "Delta=True");
mSent.Headers.Add("User-Agent", " Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.120 Safari/537.36");
try
{
mSent.Content.Headers.ContentType = new Windows.Web.Http.Headers.HttpMediaTypeHeaderValue("application/x-www-form-urlencoded");
}
catch (Exception ex) { ex.ToString(); }
//mSent.Headers.Add("Accept", "*/*");
//mSent.Headers.Add("Referer", "https://academicos.ubi.pt/online/horarios.aspx");
//mSent.Headers.Add("Accept-Encoding", "gzip,deflate");
//mSent.Headers.Add("Accept-Language", "pt-PT,pt;q=0.8,en-US;q=0.6,en;q=0.4");
mSent.Headers.Add("Cookie", "the cookie string is big, so I will not post it here");
mSent.Content = new HttpStringContent("the content is well defined, but I will not post it here it's huge"), Windows.Storage.Streams.UnicodeEncoding.Utf8);
HttpResponseMessage mReceived = await wp.SendRequestAsync(mSent, HttpCompletionOption.ResponseContentRead);
if (mReceived.IsSuccessStatusCode)
{
htmlPage = await mReceived.Content.ReadAsStringAsync();
}
}
The error that I'm recieving is Object reference not set to an instance of an object.
I tryied setting this header like I set the user agent it gives me another exception that says that to set the content type I need to set it under content headers...
Any Ideias? I tryied searching for answers for this problem, so far I came out empty handed
Pedro, you need to set the Content property to something, e.g.:
HttpRequestMessage mSent = new HttpRequestMessage(
HttpMethod.Post,
new Uri(url));
mSent.Content = new HttpStringContent(
"Name=Jonathan+Doe&Age=23",
UnicodeEncoding.Utf8,
"application/x-www-form-urlencoded");
There are other kinds of IHttpContent, such as HttpBufferContent, HttpFormUrlEncodedContent, HttpMultipartContent, HttpMultipartFormDataContent and HttpStreamContent.
You do not set the headers on the .Content member. You chould instead use the Multi-Part Content object and set the content similar to this:
HttpMultipartFormDataContent form = new HttpMultipartFormDataContent();
form.Add(new HttpStringContent(RequestBodyField.Text), "data");
HttpResponseMessage response = await httpClient.PostAsync(resourceAddress, form).AsTask(cts.Token);
Ref: http://code.msdn.microsoft.com/windowsapps/HttpClient-sample-55700664/sourcecode?fileId=98924&pathId=1116044733