I am attempting to make a program that automatically downloads some files from a Wordpress site but the problem is you have to be authenticated to download them. I have looked around but can't find anything that will really help me understand this.
This is the code that does the logging in but from what I am gathering it is not storing all the needed cookies for Wordpress, but I haven't found many examples regarding this.
var baseAddress = new Uri("https://www.gplvault.com/wp-admin");
var cookieContainer = new CookieContainer();
using (var handler = new HttpClientHandler() { CookieContainer = cookieContainer, AllowAutoRedirect = false})
using (var client = new HttpClient(handler, false) { BaseAddress = baseAddress })
{
client.DefaultRequestHeaders.Add("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36");
client.DefaultRequestHeaders.Add("Referer", "https://www.gplvault.com/my-account/");
client.DefaultRequestHeaders.Add("Connection", "keep-alive");
client.DefaultRequestHeaders.Add("Accept-Language", "en-US,en;q=0.5");
client.DefaultRequestHeaders.Add("Accept", "application/json, text/javascript, */*; q=0.01");
var homePageResult = client.GetAsync("/");
Console.WriteLine("1: " + cookieContainer.GetCookieHeader(baseAddress));
homePageResult.Result.EnsureSuccessStatusCode();
var content = new FormUrlEncodedContent(new[]{
new KeyValuePair<string, string>("username", username),
new KeyValuePair<string, string>("password", password),
});
var loginResult = client.PostAsync("/my-account/orders/", content).Result;
}
I used tamper to get the headers and this is what I got:
wget
--header="Host: www.gplvault.com"
--header="User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"
--header="Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8"
--header="Accept-Language: en-US,en;q=0.9"
--header="Cookie: __cfduid=dad983khm234b5jh23s39421e59ce2d1534278105; _ga=GA1.2.1355145636.1534278108; _gid=GA1.2.858287631.1577566529; wordpress_logged_in_13cb68234k56h43k56hk34g22133fe5=some.user%7C1535739391%7CZMz24UN1y2XPWpa1b22yfDi456fdgDGFdfg5jFufjlyKr54%7C68b0b014c9ba2dba566aa0beb44a99f60f97869dFGdfs11125361b3a2e908cd5; wfwaf-authcookie-f227c3e58kjhaf97s98dfasd9081b332=789%7Csubscriber%7Cedbc35748c970a9d4a441209cfc01e5283d791968a1cc28aa740424c08bca223; woocommerce_items_in_cart=1; woocommerce_cart_hash=faece8037500a234545305461c8ca026; wp_woocommerce_session_13cb682d9769DFGsdsb449b7722133fe5=789%7C%7C1535739438%7C%7C1535735838%7C%7C7e56398513f91b876s890d4d2f8ccff3b; _gat=1"
--header="Connection: keep-alive" "https://www.gplvault.com/download/21377/" -O "codecanyon-9899552-actionable-google-analytics-for-woocommerce.zip" -c
Related
I did a batch that parse html page of gearbest.com to extract data of the items (example link link).
It worked until 2-3 week ago after that the site was updated.
So I can't dowload pages to parse and I don't undastand why.
Before the update I did request with the following code with HtmlAgilityPack.
HtmlWeb web = new HtmlWeb();
HtmlDocument doc = null;
doc = web.Load(url); //now this the point where is throw the exception
I tried without the framework and I added some date to the request
HttpWebRequest request = (HttpWebRequest) WebRequest.Create("https://it.gearbest.com/tv-box/pp_009940949913.html");
request.Credentials = CredentialCache.DefaultCredentials;
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36";
request.ContentType = "text/html; charset=UTF-8";
request.CookieContainer = new CookieContainer();
request.Headers.Add("accept-language", "it-IT,it;q=0.9,en-US;q=0.8,en;q=0.7");
request.Headers.Add("accept-encoding", "gzip, deflate, br");
request.Headers.Add("upgrade-insecure-requests", "1");
request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8";
request.CookieContainer = new CookieContainer();
Response response = request.GetResponse(); //exception
the exception is:
IOException: Unable to read data from the transport connection
SocketException: The connection could not be established.
If I try to request the main page (https://it.gearbest.com) it works.
What's the problem in your opinion?
For some reason it doesn't like the provided user agent. If you omit setting UserAgent everything works fine
HttpWebRequest request = (HttpWebRequest) WebRequest.Create("https://it.gearbest.com/tv-box/pp_009940949913.html");
request.Credentials = CredentialCache.DefaultCredentials;
//request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36";
request.ContentType = "text/html; charset=UTF-8";
Another solution would be setting request.Connection to a random string (but not keep-alive or close)
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36";
request.Connection = "random value";
It also works but I cannot explain why.
Might be worth a try...
HttpRequest.KeepAlive = false;
HttpRequest.ProtocolVersion = HttpVersion.Version10;
https://stackoverflow.com/a/16140621/1302730
I cant force to work HttpClient Post request...
or rather, I can't understand what I'm doing wrong..
My code below and this's sketch
#region Method 2
HttpClient h = new HttpClient();
var values = new Dictionary<string, string>
{
//{ "csrf_token" , "aacc347b22df2a7b3a20c9674ba59cf4279c8d16:1552967983844" },
//{ "track_id", "501c86f19b745703e9ca54991e90dc3a87" },
//{ "password", "Oguhokuu13" }
};
CookieContainer container = new CookieContainer();
h.DefaultRequestHeaders.Add("Accept", "application/json, text/javascript, */*; q=0.01");
h.DefaultRequestHeaders.Add("Accept-Encoding", "gzip, deflate, br");
h.DefaultRequestHeaders.Add("Accept-Language", "ru-RU,ru;q=0.9,en-US;q=0.8,en;q=0.7");
h.DefaultRequestHeaders.Add("Connection", "keep-alive");
h.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/x-www-form-urlencoded")); // Content-Type
h.DefaultRequestHeaders.AcceptCharset.Add(new StringWithQualityHeaderValue("UTF-8"));
h.DefaultRequestHeaders.Add("Origin", "https://passport.yandex.ru");
h.DefaultRequestHeaders.Add("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36");
// h.DefaultRequestHeaders.Add("Referer", "https://passport.yandex.ru/auth/welcome?retpath=https%3A%2F%2Ftaxi-cabinet.mobile.yandex.ru%2Fnews%2F");
h.DefaultRequestHeaders.Add("Host", "passport.yandex.ru");
h.DefaultRequestHeaders.Add("X-Requested-With", "XMLHttpRequest");
var content = new FormUrlEncodedContent(values);
var response = h.PostAsync("https://passport.yandex.ru/auth/add", content);
response.Wait();
var responseString = response.Result.Content.ReadAsStringAsync().Result;
textBox1.Text = responseString.ToString();
#endregion
Then I get response:
\u001f?\b\0\0\0\0\0\0\u0003sN???/Q\b?\u000f\u000eQ?O,-??OLI?\u0002\0d(,h\u0016\0\0\0"
Idk what I'm doing wrong
I have a problem using the response of HttpRequest() i get the response but just the html not the headers and the key that i am searching is on the header so this is my code
HttpRequest rq = new HttpRequest();
rq.Cookies = new CookieDictionary();
rq.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36";
rq.AllowAutoRedirect = true;
rq.IgnoreProtocolErrors = true;
rq.ConnectTimeout = TimeOut;
rq.KeepAlive = true;
var str = rq.Get("url").ToString();
if(str.Contains("404")){
}
i hope you can help me
I Found the answer thanks for your help
var req = rq.Get("url");
if(req.StatusCode.ToString().Contains("NotFound") ){
}
I am trying to get info from a website that has no public API. I can't give the website's name because you literally can only sign in with a work account created on their end, so there's no way for other people to test this without my credentials (which I am not giving out).
class Client
{
public string PHPSESSID { get; set; }
public Client()
{
this.Login();
}
public void Login()
{
string csrf;
RestClient client;
RestRequest csrfRequest = new RestRequest(Method.GET);
csrfRequest.AddHeader("accept-language", "en-US,en;q=0.8");
csrfRequest.AddHeader("accept-encoding", "gzip, deflate, br");
csrfRequest.AddHeader("dnt", "1");
csrfRequest.AddHeader("accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8");
csrfRequest.AddHeader("user-agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36");
csrfRequest.AddHeader("upgrade-insecure-csrfRequests", "1");
RestRequest request = new RestRequest(Method.POST);
request.AddHeader("x-requested-with", "XMLHttpRequest");
request.AddHeader("user-agent", "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36");
request.AddHeader("referer", "{WEBSITE}/login");
request.AddHeader("origin", "{WEBSITE}");
request.AddHeader("host", "{WEBSITE}");
request.AddHeader("dnt", "1");
request.AddHeader("content-type", "application/x-www-form-urlencoded");
request.AddHeader("content-length", "118");
request.AddHeader("accept-language", "en-US,en;q=0.8");
request.AddHeader("accept-encoding", "gzip, deflate, br");
request.AddHeader("accept", "application/json, text/javascript, */*; q=0.01");
IRestResponse response;
do
{
client = new RestClient("{WEBSITE}/login");
response = client.Execute(csrfRequest);
if (String.IsNullOrEmpty(this.PHPSESSID))
this.setPHPID(response);
csrf = this.FindCSRF(response.Content);
client = new RestClient("{WEBSITE}/login_check");
RestRequest bak = request.Clone();
request.AddCookie("PHPSESSID", this.PHPSESSID);
request.AddParameter("application/x-www-form-urlencoded", "form%5Busername%5D={USERNAME}&form%5Bpassword%5D={PASSWORD}&_csrf_token=" + csrf, ParameterType.RequestBody);
client.FollowRedirects = true;
response = client.Execute(request);
request = bak.Clone();
} while (response.Content.Contains("error"));
}
public string FindCSRF(string input)
{
int find = input.IndexOf("value=", input.IndexOf("_token")) + 8;
int find2 = input.IndexOf('"', find);
--find;
return input.Substring(find, find2-find);
}
public void setPHPID(IRestResponse response)
{
try
{
string cookie = response.Headers.ToList().Find(p => p.Name.Contains("Set-Cookie")).Value.ToString();
if(String.IsNullOrEmpty(cookie))
this.PHPSESSID = response.Cookies[0].Value;
else
{
int cookieStart = cookie.IndexOf("PHPSESSID=") + 10;
this.PHPSESSID = cookie.Substring(cookieStart, cookie.IndexOf(';', cookieStart) - cookieStart);
}
}
catch
{
this.PHPSESSID = response.Cookies[0].Value;
}
}
public void switchBranch(string branchID)
{
RestRequest request = new RestRequest(Method.GET);
request.AddHeader("accept-language", "en-US,en;q=0.8");
request.AddHeader("accept-encoding", "gzip, deflate, br");
request.AddHeader("referer", "{WEBSITE}/message-board");
request.AddHeader("dnt", "1");
request.AddHeader("accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8");
request.AddHeader("user-agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36");
request.AddHeader("upgrade-insecure-requests", "1");
RestRequest branchRequest = new RestRequest(Method.POST);
branchRequest.AddHeader("accept-language", "en-US,en;q=0.8");
branchRequest.AddHeader("accept-encoding", "gzip, deflate, br");
branchRequest.AddHeader("referer", "{WEBSITE}/message-board");
branchRequest.AddHeader("dnt", "1");
branchRequest.AddHeader("content-type", "application/x-www-form-urlencoded");
branchRequest.AddHeader("user-agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36");
branchRequest.AddHeader("x-branchRequested-with", "XMLHttpbranchRequest");
branchRequest.AddHeader("origin", "{WEBSITE}");
branchRequest.AddHeader("accept", "application/json, text/javascript, */*; q=0.01");
IRestResponse response;
RestClient client;
string csrf;
do
{
client = new RestClient("{WEBSITE}/");
client.FollowRedirects = true;
RestRequest acc = request.Clone();
acc.AddCookie("PHPSESSID", this.PHPSESSID);
response = client.Execute(acc);
if (response.Content.Contains("login"))
{
this.Login();
Console.WriteLine("Login Required");
continue;
}
csrf = this.FindCSRF(response.Content);
client = new RestClient("{WEBSITE}/branch-switch");
acc = branchRequest.Clone();
acc.AddCookie("PHPSESSID", this.PHPSESSID);
acc.AddParameter("application/x-www-form-urlencoded", "office_select%5Boffice%5D=" + branchID + "&office_select%5B_token%5D=" + csrf, ParameterType.RequestBody);
} while (response.Content.Contains("error"));
}
}
public static class RequestExt
{
public static RestRequest Clone(this RestRequest req)
{
return req;
}
}
A lot of this I just intercepted from the website using Postman then copied the auto-generated code over, so if there are certain headers that can help, I probably don't know about them.
I mostly fixed the login issues (though I literally have to fail before it gives me a working CSRF token, thus the while loops). The problem is that when I try to go to {WEBSITE} in Client.switchBranch, to start actually doing stuff, it returns a JSON with a redirect to the login page. It doesn't do this in Postman, I have been able to successfully interact with the website and gather information... once or twice (logging in even sometimes fails in the browser).
It seems entirely random to me, but there MUST be some kind of common factor here. I'm guessing it's something to do with the PHPSESSID cookie or the lack of a "keep-alive" connection header (which caused RestSharp to throw an error when I tried to implement it; even if that is the problem, I still don't know how to solve it). The login request and response headers in Chrome's developer tools, intercepting from the website, look something like this.
I am trying to capture the return html after a button click. I've used fiddler to see what request gets sent when the user clicks a button. Below is the header information.
GET http://www.nseindia.com/products/dynaContent/equities/equities/bulkdeals.jsp?symbol=&segmentLink=13&symbolCount=&dateRange=day&fromDate=&toDate=&dataType=DEALS HTTP/1.1
Host: www.nseindia.com
Proxy-Connection: keep-alive
Accept: */*
X-Requested-With: XMLHttpRequest
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36
Referer: http://www.nseindia.com/products/content/equities/equities/bulk.htm
Accept-Encoding: gzip, deflate, sdch
Accept-Language: en-US,en;q=0.8
I have the below code which is not working. Can someone please point me to the correct solution ?
var nseBulkDealsUrl = #"http://www.nseindia.com/products/dynaContent/equities/equities/bulkdeals.jsp?symbol=&segmentLink=13&symbolCount=&dateRange=day&fromDate=&toDate=&dataType=DEALS";
var client = new WebClient();
client.Encoding = Encoding.UTF8;
var values = new NameValueCollection();
values.Add("Referer", "http://www.nseindia.com/products/content/equities/equities/bulk.htm");
values.Add("User-Agent", "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36");
values.Add("X-Requested-With", "XMLHttpRequest");
client.Headers.Add(values);
client.Proxy = WebRequest.DefaultWebProxy;
var htmlPageSource = client.DownloadString(nseBulkDealsUrl);
You have to fill enough information for headers.
It works with this code:
var nseBulkDealsUrl = #"http://www.nseindia.com/products/dynaContent/equities/equities/bulkdeals.jsp?symbol=&segmentLink=13&symbolCount=&dateRange=day&fromDate=&toDate=&dataType=DEALS";
var client = new WebClient();
client.Encoding = Encoding.UTF8;
var values = new NameValueCollection();
values.Add("Referer", "http://www.nseindia.com/products/content/equities/equities/bulk.htm");
values.Add("User-Agent", "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36");
values.Add("X-Requested-With", "XMLHttpRequest");
values.Add("Accept", "*/*");
//values.Add("Accept-Encoding", "gzip, deflate, sdch");
values.Add("Accept-Language", "en-US,en;q=0.8");
client.Proxy = WebRequest.DefaultWebProxy;
//client.UploadValues(nseBulkDealsUrl, "GET", values);
client.Headers.Add(values);
var htmlPageSource = client.DownloadString(nseBulkDealsUrl);