I am attempting to make a program that automatically downloads some files from a Wordpress site but the problem is you have to be authenticated to download them. I have looked around but can't find anything that will really help me understand this.
This is the code that does the logging in but from what I am gathering it is not storing all the needed cookies for Wordpress, but I haven't found many examples regarding this.
var baseAddress = new Uri("https://www.gplvault.com/wp-admin");
var cookieContainer = new CookieContainer();
using (var handler = new HttpClientHandler() { CookieContainer = cookieContainer, AllowAutoRedirect = false})
using (var client = new HttpClient(handler, false) { BaseAddress = baseAddress })
{
client.DefaultRequestHeaders.Add("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36");
client.DefaultRequestHeaders.Add("Referer", "https://www.gplvault.com/my-account/");
client.DefaultRequestHeaders.Add("Connection", "keep-alive");
client.DefaultRequestHeaders.Add("Accept-Language", "en-US,en;q=0.5");
client.DefaultRequestHeaders.Add("Accept", "application/json, text/javascript, */*; q=0.01");
var homePageResult = client.GetAsync("/");
Console.WriteLine("1: " + cookieContainer.GetCookieHeader(baseAddress));
homePageResult.Result.EnsureSuccessStatusCode();
var content = new FormUrlEncodedContent(new[]{
new KeyValuePair<string, string>("username", username),
new KeyValuePair<string, string>("password", password),
});
var loginResult = client.PostAsync("/my-account/orders/", content).Result;
}
I used tamper to get the headers and this is what I got:
wget
--header="Host: www.gplvault.com"
--header="User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"
--header="Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8"
--header="Accept-Language: en-US,en;q=0.9"
--header="Cookie: __cfduid=dad983khm234b5jh23s39421e59ce2d1534278105; _ga=GA1.2.1355145636.1534278108; _gid=GA1.2.858287631.1577566529; wordpress_logged_in_13cb68234k56h43k56hk34g22133fe5=some.user%7C1535739391%7CZMz24UN1y2XPWpa1b22yfDi456fdgDGFdfg5jFufjlyKr54%7C68b0b014c9ba2dba566aa0beb44a99f60f97869dFGdfs11125361b3a2e908cd5; wfwaf-authcookie-f227c3e58kjhaf97s98dfasd9081b332=789%7Csubscriber%7Cedbc35748c970a9d4a441209cfc01e5283d791968a1cc28aa740424c08bca223; woocommerce_items_in_cart=1; woocommerce_cart_hash=faece8037500a234545305461c8ca026; wp_woocommerce_session_13cb682d9769DFGsdsb449b7722133fe5=789%7C%7C1535739438%7C%7C1535735838%7C%7C7e56398513f91b876s890d4d2f8ccff3b; _gat=1"
--header="Connection: keep-alive" "https://www.gplvault.com/download/21377/" -O "codecanyon-9899552-actionable-google-analytics-for-woocommerce.zip" -c
I have a problem using the response of HttpRequest() i get the response but just the html not the headers and the key that i am searching is on the header so this is my code
HttpRequest rq = new HttpRequest();
rq.Cookies = new CookieDictionary();
rq.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36";
rq.AllowAutoRedirect = true;
rq.IgnoreProtocolErrors = true;
rq.ConnectTimeout = TimeOut;
rq.KeepAlive = true;
var str = rq.Get("url").ToString();
if(str.Contains("404")){
}
i hope you can help me
I Found the answer thanks for your help
var req = rq.Get("url");
if(req.StatusCode.ToString().Contains("NotFound") ){
}
I am trying to get info from a website that has no public API. I can't give the website's name because you literally can only sign in with a work account created on their end, so there's no way for other people to test this without my credentials (which I am not giving out).
class Client
{
public string PHPSESSID { get; set; }
public Client()
{
this.Login();
}
public void Login()
{
string csrf;
RestClient client;
RestRequest csrfRequest = new RestRequest(Method.GET);
csrfRequest.AddHeader("accept-language", "en-US,en;q=0.8");
csrfRequest.AddHeader("accept-encoding", "gzip, deflate, br");
csrfRequest.AddHeader("dnt", "1");
csrfRequest.AddHeader("accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8");
csrfRequest.AddHeader("user-agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36");
csrfRequest.AddHeader("upgrade-insecure-csrfRequests", "1");
RestRequest request = new RestRequest(Method.POST);
request.AddHeader("x-requested-with", "XMLHttpRequest");
request.AddHeader("user-agent", "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36");
request.AddHeader("referer", "{WEBSITE}/login");
request.AddHeader("origin", "{WEBSITE}");
request.AddHeader("host", "{WEBSITE}");
request.AddHeader("dnt", "1");
request.AddHeader("content-type", "application/x-www-form-urlencoded");
request.AddHeader("content-length", "118");
request.AddHeader("accept-language", "en-US,en;q=0.8");
request.AddHeader("accept-encoding", "gzip, deflate, br");
request.AddHeader("accept", "application/json, text/javascript, */*; q=0.01");
IRestResponse response;
do
{
client = new RestClient("{WEBSITE}/login");
response = client.Execute(csrfRequest);
if (String.IsNullOrEmpty(this.PHPSESSID))
this.setPHPID(response);
csrf = this.FindCSRF(response.Content);
client = new RestClient("{WEBSITE}/login_check");
RestRequest bak = request.Clone();
request.AddCookie("PHPSESSID", this.PHPSESSID);
request.AddParameter("application/x-www-form-urlencoded", "form%5Busername%5D={USERNAME}&form%5Bpassword%5D={PASSWORD}&_csrf_token=" + csrf, ParameterType.RequestBody);
client.FollowRedirects = true;
response = client.Execute(request);
request = bak.Clone();
} while (response.Content.Contains("error"));
}
public string FindCSRF(string input)
{
int find = input.IndexOf("value=", input.IndexOf("_token")) + 8;
int find2 = input.IndexOf('"', find);
--find;
return input.Substring(find, find2-find);
}
public void setPHPID(IRestResponse response)
{
try
{
string cookie = response.Headers.ToList().Find(p => p.Name.Contains("Set-Cookie")).Value.ToString();
if(String.IsNullOrEmpty(cookie))
this.PHPSESSID = response.Cookies[0].Value;
else
{
int cookieStart = cookie.IndexOf("PHPSESSID=") + 10;
this.PHPSESSID = cookie.Substring(cookieStart, cookie.IndexOf(';', cookieStart) - cookieStart);
}
}
catch
{
this.PHPSESSID = response.Cookies[0].Value;
}
}
public void switchBranch(string branchID)
{
RestRequest request = new RestRequest(Method.GET);
request.AddHeader("accept-language", "en-US,en;q=0.8");
request.AddHeader("accept-encoding", "gzip, deflate, br");
request.AddHeader("referer", "{WEBSITE}/message-board");
request.AddHeader("dnt", "1");
request.AddHeader("accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8");
request.AddHeader("user-agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36");
request.AddHeader("upgrade-insecure-requests", "1");
RestRequest branchRequest = new RestRequest(Method.POST);
branchRequest.AddHeader("accept-language", "en-US,en;q=0.8");
branchRequest.AddHeader("accept-encoding", "gzip, deflate, br");
branchRequest.AddHeader("referer", "{WEBSITE}/message-board");
branchRequest.AddHeader("dnt", "1");
branchRequest.AddHeader("content-type", "application/x-www-form-urlencoded");
branchRequest.AddHeader("user-agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36");
branchRequest.AddHeader("x-branchRequested-with", "XMLHttpbranchRequest");
branchRequest.AddHeader("origin", "{WEBSITE}");
branchRequest.AddHeader("accept", "application/json, text/javascript, */*; q=0.01");
IRestResponse response;
RestClient client;
string csrf;
do
{
client = new RestClient("{WEBSITE}/");
client.FollowRedirects = true;
RestRequest acc = request.Clone();
acc.AddCookie("PHPSESSID", this.PHPSESSID);
response = client.Execute(acc);
if (response.Content.Contains("login"))
{
this.Login();
Console.WriteLine("Login Required");
continue;
}
csrf = this.FindCSRF(response.Content);
client = new RestClient("{WEBSITE}/branch-switch");
acc = branchRequest.Clone();
acc.AddCookie("PHPSESSID", this.PHPSESSID);
acc.AddParameter("application/x-www-form-urlencoded", "office_select%5Boffice%5D=" + branchID + "&office_select%5B_token%5D=" + csrf, ParameterType.RequestBody);
} while (response.Content.Contains("error"));
}
}
public static class RequestExt
{
public static RestRequest Clone(this RestRequest req)
{
return req;
}
}
A lot of this I just intercepted from the website using Postman then copied the auto-generated code over, so if there are certain headers that can help, I probably don't know about them.
I mostly fixed the login issues (though I literally have to fail before it gives me a working CSRF token, thus the while loops). The problem is that when I try to go to {WEBSITE} in Client.switchBranch, to start actually doing stuff, it returns a JSON with a redirect to the login page. It doesn't do this in Postman, I have been able to successfully interact with the website and gather information... once or twice (logging in even sometimes fails in the browser).
It seems entirely random to me, but there MUST be some kind of common factor here. I'm guessing it's something to do with the PHPSESSID cookie or the lack of a "keep-alive" connection header (which caused RestSharp to throw an error when I tried to implement it; even if that is the problem, I still don't know how to solve it). The login request and response headers in Chrome's developer tools, intercepting from the website, look something like this.
I'm trying to send a GET request with HttpClient:
public static HttpClient Client { get
{
var handler = new HttpClientHandler() { AutomaticDecompression = System.Net.DecompressionMethods.Deflate | System.Net.DecompressionMethods.GZip };
var http = new HttpClient(handler) { BaseAddress = new Uri("https://www.example.com/") };
http.DefaultRequestHeaders.Accept.Clear();
http.DefaultRequestHeaders.Accept.TryParseAdd("*/*");
http.DefaultRequestHeaders.AcceptLanguage.TryParseAdd("en-US;q=0.6,en;q=0.4");
http.DefaultRequestHeaders.AcceptEncoding.TryParseAdd("gzip, deflate, br");
http.DefaultRequestHeaders.Host = "example.com";
http.DefaultRequestHeaders.Add("X-Requested-With", "XMLHttpRequest");
http.DefaultRequestHeaders.Add("Connection", "keep-alive");
http.DefaultRequestHeaders.Add("Keep-Alive", "600");
http.DefaultRequestHeaders.UserAgent.TryParseAdd("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36");
ServicePointManager
.ServerCertificateValidationCallback +=
(sender, cert, chain, sslPolicyErrors) => true;
return http;
}
...
using (var client = HttpHelper.Client)
using (var res = await client.GetAsync("api/stuff/filter?order=" + order + "&page=" + page))
if (res.IsSuccessStatusCode)
{
var json = await res.Content.ReadAsStringAsync();
var response = JsonConvert.DeserializeObject<AnimeResponse>(json);
return response;
}
Unfortunately this sends a request without the www. part and results in 301 status code as well as many, many failed redirects to the same address (without the www. part).
How can I fix this?
Edit: When I re-send the request with Fiddler it also returns 301, but the n it redirects correctly to wwww.. The HttpClient doesn't, and just redirects to the same www-less URI.
Cookies from CookieContainer are not added to Get or Post requests. Other headers work without a problem. What is the correct way to add cookies? I have done this before without a problem but I can't locate the error here.
var cookieContainer = new CookieContainer();
var handler = new HttpClientHandler();
handler.AllowAutoRedirect = true;
handler.UseCookies = true;
handler.CookieContainer = cookieContainer;
var baseAddress = new Uri("https://www.example.se");
cookieContainer.Add(baseAddress, new Cookie("Testing", "Test"));
//This did not work either
//cookieContainer.Add(baseAddress, new Cookie("Testing", "Test", "/"));
using (var client = new HttpClient(new LoggingHandler(handler)))
{
client.BaseAddress = baseAddress;
client.DefaultRequestHeaders.Add("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36");
client.DefaultRequestHeaders.Add("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8");
client.DefaultRequestHeaders.Add("Accept-Language", "sv-SE,sv;q=0.8,en-US;q=0.6,en;q=0.4");
var getResponse = client.GetAsync("/test").Result;
string responseString = getResponse.Content.ReadAsStringAsync().Result;
}
LoggingHandler:
https://stackoverflow.com/a/18925296/3850405
For some reason cookies aren't in the HttpRequestMessage headers when using CookieContainer. If I checked my CookieContainer object they were there though.
var cookieList = new List<Cookie>();
foreach (Cookie cookie in cookieContainer.GetCookies(baseAddress))
{
cookieList.Add(cookie);
}