I am trying to capture the return html after a button click. I've used fiddler to see what request gets sent when the user clicks a button. Below is the header information.
GET http://www.nseindia.com/products/dynaContent/equities/equities/bulkdeals.jsp?symbol=&segmentLink=13&symbolCount=&dateRange=day&fromDate=&toDate=&dataType=DEALS HTTP/1.1
Host: www.nseindia.com
Proxy-Connection: keep-alive
Accept: */*
X-Requested-With: XMLHttpRequest
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36
Referer: http://www.nseindia.com/products/content/equities/equities/bulk.htm
Accept-Encoding: gzip, deflate, sdch
Accept-Language: en-US,en;q=0.8
I have the below code which is not working. Can someone please point me to the correct solution ?
var nseBulkDealsUrl = #"http://www.nseindia.com/products/dynaContent/equities/equities/bulkdeals.jsp?symbol=&segmentLink=13&symbolCount=&dateRange=day&fromDate=&toDate=&dataType=DEALS";
var client = new WebClient();
client.Encoding = Encoding.UTF8;
var values = new NameValueCollection();
values.Add("Referer", "http://www.nseindia.com/products/content/equities/equities/bulk.htm");
values.Add("User-Agent", "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36");
values.Add("X-Requested-With", "XMLHttpRequest");
client.Headers.Add(values);
client.Proxy = WebRequest.DefaultWebProxy;
var htmlPageSource = client.DownloadString(nseBulkDealsUrl);
You have to fill enough information for headers.
It works with this code:
var nseBulkDealsUrl = #"http://www.nseindia.com/products/dynaContent/equities/equities/bulkdeals.jsp?symbol=&segmentLink=13&symbolCount=&dateRange=day&fromDate=&toDate=&dataType=DEALS";
var client = new WebClient();
client.Encoding = Encoding.UTF8;
var values = new NameValueCollection();
values.Add("Referer", "http://www.nseindia.com/products/content/equities/equities/bulk.htm");
values.Add("User-Agent", "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36");
values.Add("X-Requested-With", "XMLHttpRequest");
values.Add("Accept", "*/*");
//values.Add("Accept-Encoding", "gzip, deflate, sdch");
values.Add("Accept-Language", "en-US,en;q=0.8");
client.Proxy = WebRequest.DefaultWebProxy;
//client.UploadValues(nseBulkDealsUrl, "GET", values);
client.Headers.Add(values);
var htmlPageSource = client.DownloadString(nseBulkDealsUrl);
Related
I did a batch that parse html page of gearbest.com to extract data of the items (example link link).
It worked until 2-3 week ago after that the site was updated.
So I can't dowload pages to parse and I don't undastand why.
Before the update I did request with the following code with HtmlAgilityPack.
HtmlWeb web = new HtmlWeb();
HtmlDocument doc = null;
doc = web.Load(url); //now this the point where is throw the exception
I tried without the framework and I added some date to the request
HttpWebRequest request = (HttpWebRequest) WebRequest.Create("https://it.gearbest.com/tv-box/pp_009940949913.html");
request.Credentials = CredentialCache.DefaultCredentials;
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36";
request.ContentType = "text/html; charset=UTF-8";
request.CookieContainer = new CookieContainer();
request.Headers.Add("accept-language", "it-IT,it;q=0.9,en-US;q=0.8,en;q=0.7");
request.Headers.Add("accept-encoding", "gzip, deflate, br");
request.Headers.Add("upgrade-insecure-requests", "1");
request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8";
request.CookieContainer = new CookieContainer();
Response response = request.GetResponse(); //exception
the exception is:
IOException: Unable to read data from the transport connection
SocketException: The connection could not be established.
If I try to request the main page (https://it.gearbest.com) it works.
What's the problem in your opinion?
For some reason it doesn't like the provided user agent. If you omit setting UserAgent everything works fine
HttpWebRequest request = (HttpWebRequest) WebRequest.Create("https://it.gearbest.com/tv-box/pp_009940949913.html");
request.Credentials = CredentialCache.DefaultCredentials;
//request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36";
request.ContentType = "text/html; charset=UTF-8";
Another solution would be setting request.Connection to a random string (but not keep-alive or close)
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36";
request.Connection = "random value";
It also works but I cannot explain why.
Might be worth a try...
HttpRequest.KeepAlive = false;
HttpRequest.ProtocolVersion = HttpVersion.Version10;
https://stackoverflow.com/a/16140621/1302730
I am trying to download a pdf file from a web server over https in c# and getting HTTP OK response but file stream is not coming with response. What am I doing wrong?
Here is my code behind:
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
ServicePointManager.ServerCertificateValidationCallback = (s, cert, ch, sec) => { return true; };
HttpWebRequest myRequest = (HttpWebRequest)HttpWebRequest.Create("https://services.dps.ohio.gov/CrashOnline/CrashRetrieval/OhioCrashReportRetrieval/GetReport");
myRequest.Method = "POST";
myRequest.CookieContainer = cookies;
myRequest.ContentType = "application/x-www-form-urlencoded";
myRequest.Accept = myRequest.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8";
myRequest.Headers.Add(HttpRequestHeader.AcceptEncoding, "gzip, deflate, br");
myRequest.Headers.Add(HttpRequestHeader.AcceptLanguage, "en-US,en;q=0.9");
myRequest.Referer = "https://services.dps.ohio.gov/CrashOnline/CrashRetrieval/OhioCrashReportRetrieval/Search";
myRequest.UserAgent = "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36";
byte[] bytes = Encoding.Default.GetBytes(content);
myRequest.ContentLength = bytes.Length;
Stream dataStream = myRequest.GetRequestStream();
dataStream.Write(bytes, 0, bytes.Length);
dataStream.Close();
byte[] result = null;
byte[] buffer = new byte[2048];
using (HttpWebResponse response = (HttpWebResponse)myRequest.GetResponse())
Downloading file from chrome is working fine and here is fiddler RAW http packet from browser:
POST https://services.dps.ohio.gov/CrashOnline/CrashRetrieval/OhioCrashReportRetrieval/GetReport HTTP/1.1
Host: services.dps.ohio.gov
Connection: keep-alive
Content-Length: 177
Cache-Control: max-age=0
Origin: https://services.dps.ohio.gov
Upgrade-Insecure-Requests: 1
Content-Type: application/x-www-form-urlencoded
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
Referer: https://services.dps.ohio.gov/CrashOnline/CrashRetrieval/OhioCrashReportRetrieval/Search
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9
Cookie: ASP.NET_SessionId=t1nddo51g0jotl51cqev15rp; __RequestVerificationToken_L0NyYXNoT25saW5l0=iMy0lJ3OviGs0iXame8ebp8IdS6K9wpc8tFinYd0nhNnySTLlii-yHYU5tDJjv49Fqi29M_31mPiPdKWWtJJPWSmp4qGGvna9lW2rkDHUAs1
__RequestVerificationToken=eTJN31araRPeB-bhQaeU3awBDsCwR9BpkIYNqpsakTPRT1w1nXtbGFcfVRZS776ZW0NGv1aKahx3WJv5PNQ199iM7GF9Y9f7ws5KTG4gI3I1&id=NjgzMTIwOQ%3D%3D&btn_NjgzMTIwOQ%3D%3D=
This is my application's http packet:
POST https://services.dps.ohio.gov/CrashOnline/CrashRetrieval/OhioCrashReportRetrieval/GetReport HTTP/1.1
Accept: text/html, application/xhtml+xml, image/jxr, */*
Referer: https://services.dps.ohio.gov/CrashOnline/CrashRetrieval/OhioCrashReportRetrieval/Search
Accept-Language: en-GB,es-ES;q=0.8,es;q=0.6,en-US;q=0.4,en;q=0.2
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko
Content-Type: application/x-www-form-urlencoded
Accept-Encoding: gzip, deflate
Host: services.dps.ohio.gov
Content-Length: 177
Connection: Keep-Alive
Cache-Control: no-cache
__RequestVerificationToken=eTJN31araRPeB-bhQaeU3awBDsCwR9BpkIYNqpsakTPRT1w1nXtbGFcfVRZS776ZW0NGv1aKahx3WJv5PNQ199iM7GF9Y9f7ws5KTG4gI3I1&id=NjgzMTIwOQ%3D%3D&btn_NjgzMTIwOQ%3D%3D=
I am attempting to make a program that automatically downloads some files from a Wordpress site but the problem is you have to be authenticated to download them. I have looked around but can't find anything that will really help me understand this.
This is the code that does the logging in but from what I am gathering it is not storing all the needed cookies for Wordpress, but I haven't found many examples regarding this.
var baseAddress = new Uri("https://www.gplvault.com/wp-admin");
var cookieContainer = new CookieContainer();
using (var handler = new HttpClientHandler() { CookieContainer = cookieContainer, AllowAutoRedirect = false})
using (var client = new HttpClient(handler, false) { BaseAddress = baseAddress })
{
client.DefaultRequestHeaders.Add("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36");
client.DefaultRequestHeaders.Add("Referer", "https://www.gplvault.com/my-account/");
client.DefaultRequestHeaders.Add("Connection", "keep-alive");
client.DefaultRequestHeaders.Add("Accept-Language", "en-US,en;q=0.5");
client.DefaultRequestHeaders.Add("Accept", "application/json, text/javascript, */*; q=0.01");
var homePageResult = client.GetAsync("/");
Console.WriteLine("1: " + cookieContainer.GetCookieHeader(baseAddress));
homePageResult.Result.EnsureSuccessStatusCode();
var content = new FormUrlEncodedContent(new[]{
new KeyValuePair<string, string>("username", username),
new KeyValuePair<string, string>("password", password),
});
var loginResult = client.PostAsync("/my-account/orders/", content).Result;
}
I used tamper to get the headers and this is what I got:
wget
--header="Host: www.gplvault.com"
--header="User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"
--header="Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8"
--header="Accept-Language: en-US,en;q=0.9"
--header="Cookie: __cfduid=dad983khm234b5jh23s39421e59ce2d1534278105; _ga=GA1.2.1355145636.1534278108; _gid=GA1.2.858287631.1577566529; wordpress_logged_in_13cb68234k56h43k56hk34g22133fe5=some.user%7C1535739391%7CZMz24UN1y2XPWpa1b22yfDi456fdgDGFdfg5jFufjlyKr54%7C68b0b014c9ba2dba566aa0beb44a99f60f97869dFGdfs11125361b3a2e908cd5; wfwaf-authcookie-f227c3e58kjhaf97s98dfasd9081b332=789%7Csubscriber%7Cedbc35748c970a9d4a441209cfc01e5283d791968a1cc28aa740424c08bca223; woocommerce_items_in_cart=1; woocommerce_cart_hash=faece8037500a234545305461c8ca026; wp_woocommerce_session_13cb682d9769DFGsdsb449b7722133fe5=789%7C%7C1535739438%7C%7C1535735838%7C%7C7e56398513f91b876s890d4d2f8ccff3b; _gat=1"
--header="Connection: keep-alive" "https://www.gplvault.com/download/21377/" -O "codecanyon-9899552-actionable-google-analytics-for-woocommerce.zip" -c
I am trying to get info from a website that has no public API. I can't give the website's name because you literally can only sign in with a work account created on their end, so there's no way for other people to test this without my credentials (which I am not giving out).
class Client
{
public string PHPSESSID { get; set; }
public Client()
{
this.Login();
}
public void Login()
{
string csrf;
RestClient client;
RestRequest csrfRequest = new RestRequest(Method.GET);
csrfRequest.AddHeader("accept-language", "en-US,en;q=0.8");
csrfRequest.AddHeader("accept-encoding", "gzip, deflate, br");
csrfRequest.AddHeader("dnt", "1");
csrfRequest.AddHeader("accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8");
csrfRequest.AddHeader("user-agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36");
csrfRequest.AddHeader("upgrade-insecure-csrfRequests", "1");
RestRequest request = new RestRequest(Method.POST);
request.AddHeader("x-requested-with", "XMLHttpRequest");
request.AddHeader("user-agent", "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36");
request.AddHeader("referer", "{WEBSITE}/login");
request.AddHeader("origin", "{WEBSITE}");
request.AddHeader("host", "{WEBSITE}");
request.AddHeader("dnt", "1");
request.AddHeader("content-type", "application/x-www-form-urlencoded");
request.AddHeader("content-length", "118");
request.AddHeader("accept-language", "en-US,en;q=0.8");
request.AddHeader("accept-encoding", "gzip, deflate, br");
request.AddHeader("accept", "application/json, text/javascript, */*; q=0.01");
IRestResponse response;
do
{
client = new RestClient("{WEBSITE}/login");
response = client.Execute(csrfRequest);
if (String.IsNullOrEmpty(this.PHPSESSID))
this.setPHPID(response);
csrf = this.FindCSRF(response.Content);
client = new RestClient("{WEBSITE}/login_check");
RestRequest bak = request.Clone();
request.AddCookie("PHPSESSID", this.PHPSESSID);
request.AddParameter("application/x-www-form-urlencoded", "form%5Busername%5D={USERNAME}&form%5Bpassword%5D={PASSWORD}&_csrf_token=" + csrf, ParameterType.RequestBody);
client.FollowRedirects = true;
response = client.Execute(request);
request = bak.Clone();
} while (response.Content.Contains("error"));
}
public string FindCSRF(string input)
{
int find = input.IndexOf("value=", input.IndexOf("_token")) + 8;
int find2 = input.IndexOf('"', find);
--find;
return input.Substring(find, find2-find);
}
public void setPHPID(IRestResponse response)
{
try
{
string cookie = response.Headers.ToList().Find(p => p.Name.Contains("Set-Cookie")).Value.ToString();
if(String.IsNullOrEmpty(cookie))
this.PHPSESSID = response.Cookies[0].Value;
else
{
int cookieStart = cookie.IndexOf("PHPSESSID=") + 10;
this.PHPSESSID = cookie.Substring(cookieStart, cookie.IndexOf(';', cookieStart) - cookieStart);
}
}
catch
{
this.PHPSESSID = response.Cookies[0].Value;
}
}
public void switchBranch(string branchID)
{
RestRequest request = new RestRequest(Method.GET);
request.AddHeader("accept-language", "en-US,en;q=0.8");
request.AddHeader("accept-encoding", "gzip, deflate, br");
request.AddHeader("referer", "{WEBSITE}/message-board");
request.AddHeader("dnt", "1");
request.AddHeader("accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8");
request.AddHeader("user-agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36");
request.AddHeader("upgrade-insecure-requests", "1");
RestRequest branchRequest = new RestRequest(Method.POST);
branchRequest.AddHeader("accept-language", "en-US,en;q=0.8");
branchRequest.AddHeader("accept-encoding", "gzip, deflate, br");
branchRequest.AddHeader("referer", "{WEBSITE}/message-board");
branchRequest.AddHeader("dnt", "1");
branchRequest.AddHeader("content-type", "application/x-www-form-urlencoded");
branchRequest.AddHeader("user-agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36");
branchRequest.AddHeader("x-branchRequested-with", "XMLHttpbranchRequest");
branchRequest.AddHeader("origin", "{WEBSITE}");
branchRequest.AddHeader("accept", "application/json, text/javascript, */*; q=0.01");
IRestResponse response;
RestClient client;
string csrf;
do
{
client = new RestClient("{WEBSITE}/");
client.FollowRedirects = true;
RestRequest acc = request.Clone();
acc.AddCookie("PHPSESSID", this.PHPSESSID);
response = client.Execute(acc);
if (response.Content.Contains("login"))
{
this.Login();
Console.WriteLine("Login Required");
continue;
}
csrf = this.FindCSRF(response.Content);
client = new RestClient("{WEBSITE}/branch-switch");
acc = branchRequest.Clone();
acc.AddCookie("PHPSESSID", this.PHPSESSID);
acc.AddParameter("application/x-www-form-urlencoded", "office_select%5Boffice%5D=" + branchID + "&office_select%5B_token%5D=" + csrf, ParameterType.RequestBody);
} while (response.Content.Contains("error"));
}
}
public static class RequestExt
{
public static RestRequest Clone(this RestRequest req)
{
return req;
}
}
A lot of this I just intercepted from the website using Postman then copied the auto-generated code over, so if there are certain headers that can help, I probably don't know about them.
I mostly fixed the login issues (though I literally have to fail before it gives me a working CSRF token, thus the while loops). The problem is that when I try to go to {WEBSITE} in Client.switchBranch, to start actually doing stuff, it returns a JSON with a redirect to the login page. It doesn't do this in Postman, I have been able to successfully interact with the website and gather information... once or twice (logging in even sometimes fails in the browser).
It seems entirely random to me, but there MUST be some kind of common factor here. I'm guessing it's something to do with the PHPSESSID cookie or the lack of a "keep-alive" connection header (which caused RestSharp to throw an error when I tried to implement it; even if that is the problem, I still don't know how to solve it). The login request and response headers in Chrome's developer tools, intercepting from the website, look something like this.
I'm setting up some HTTP request headers like this:
this.Url = new Uri(u);
HttpWebRequest http = (HttpWebRequest)WebRequest.Create(Url);
WebResponse response = http.GetResponse();
//headers
http.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/20100101 Firefox/17.0\r\n";
http.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\n";
http.Headers.Add("Accept-Encoding", "gzip,deflate,sdch'r'n");
http.Headers.Add("Accept-Language", "en-US,en;q=0.9\r\n");
http.Headers.Add("Accept-Charset", "ISO-8859-1,utf-8;q=0.7,*;q=0.3\r\n");
I am capturing them like this:
for (count = 0; count < http.Headers.Keys.Count; count++)
{
headerKey = http.Headers.Keys[count];
headerValue = http.Headers[headerKey];
if (headerValue != null)
{
if (headerKey == null)
{
requestbuffer.Append(headerValue);
requestbuffer.Append(Newline);
}
else
{
requestbuffer.Append(headerKey + ": " + headerValue);
requestbuffer.Append(Newline);
}
}
}
When I run the testing tool everything seems good:
Host: domain.com
Connection: Keep-Alive
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0)Gecko/20100101 Firefox/17.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8
Accept-Encoding: gzip,deflate,sdch Accept-Language: en-US,en;q=0.9
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
However in Wireshark and Fiddler only the following header is sent:
GET / HTTP/1.1
Host: domain.com
Any idea why that may be?
You're trying to add the headers after you've called http.GetResponse(). That's after the request has been sent. Change it to this:
this.Url = new Uri(u);
HttpWebRequest http = (HttpWebRequest)WebRequest.Create(Url);
//headers
http.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/20100101 Firefox/17.0\r\n";
http.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\n";
http.Headers.Add("Accept-Encoding", "gzip,deflate,sdch'r'n");
http.Headers.Add("Accept-Language", "en-US,en;q=0.9\r\n");
http.Headers.Add("Accept-Charset", "ISO-8859-1,utf-8;q=0.7,*;q=0.3\r\n");
using (WebResponse response = http.GetResponse())
{
// Do whatever
}
(Note that you really should be disposing of the response.)
Yes, you are setting the header after you have sent the request.
Try this:
//headers
http.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/20100101 Firefox/17.0\r\n";
http.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\n";
http.Headers.Add("Accept-Encoding", "gzip,deflate,sdch'r'n");
http.Headers.Add("Accept-Language", "en-US,en;q=0.9\r\n");
http.Headers.Add("Accept-Charset", "ISO-8859-1,utf-8;q=0.7,*;q=0.3\r\n");
//Added diesposal of response too
using (WebResponse response = http.GetResponse())
{
}