We Keep Coding

Can't download web page in .net

I did a batch that parse html page of gearbest.com to extract data of the items (example link link).
It worked until 2-3 week ago after that the site was updated.
So I can't dowload pages to parse and I don't undastand why.
Before the update I did request with the following code with HtmlAgilityPack.
HtmlWeb web = new HtmlWeb();
HtmlDocument doc = null;
doc = web.Load(url); //now this the point where is throw the exception
I tried without the framework and I added some date to the request
HttpWebRequest request = (HttpWebRequest) WebRequest.Create("https://it.gearbest.com/tv-box/pp_009940949913.html");
request.Credentials = CredentialCache.DefaultCredentials;
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36";
request.ContentType = "text/html; charset=UTF-8";
request.CookieContainer = new CookieContainer();
request.Headers.Add("accept-language", "it-IT,it;q=0.9,en-US;q=0.8,en;q=0.7");
request.Headers.Add("accept-encoding", "gzip, deflate, br");
request.Headers.Add("upgrade-insecure-requests", "1");
request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8";
request.CookieContainer = new CookieContainer();
Response response = request.GetResponse(); //exception
the exception is:
IOException: Unable to read data from the transport connection
SocketException: The connection could not be established.
If I try to request the main page (https://it.gearbest.com) it works.
What's the problem in your opinion?

For some reason it doesn't like the provided user agent. If you omit setting UserAgent everything works fine
HttpWebRequest request = (HttpWebRequest) WebRequest.Create("https://it.gearbest.com/tv-box/pp_009940949913.html");
request.Credentials = CredentialCache.DefaultCredentials;
//request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36";
request.ContentType = "text/html; charset=UTF-8";
Another solution would be setting request.Connection to a random string (but not keep-alive or close)
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36";
request.Connection = "random value";
It also works but I cannot explain why.

Might be worth a try...
HttpRequest.KeepAlive = false;
HttpRequest.ProtocolVersion = HttpVersion.Version10;
https://stackoverflow.com/a/16140621/1302730

C# htmlagility, getting exception when i add header in following code

I'm getting exception when i run this code
Exception "header must be modified using the appropriate property or method."
HtmlAgilityPack.HtmlWeb web = new HtmlWeb();
web.UserAgent="Mozilla/5.0 (Windows NT 10.0; Win64; x64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36";
web.PreRequest += (request) =>
{
request.Headers.Add("Accept","text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
request.Headers.Add("Accept-Language", "de-DE");
return true;
};
HtmlAgilityPack.HtmlDocument doc = new
HtmlAgilityPack.HtmlDocument();
doc = web.Load("http://www.alfatah.pk/");

This works for me in https://dotnetfiddle.net/AQbs3v :
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://www.alfatah.pk/");
request.UserAgent = "Mozilla / 5.0(Windows NT 10.0; Win64; x64) AppleWebKit / 537.36(KHTML, like Gecko) Chrome / 70.0.3538.77 Safari / 537.36";
request.Accept= "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
//request.Headers.Add(HttpRequestHeader.AcceptLanguage, "de-DE");
using (var response = (HttpWebResponse)(request.GetResponse()))
{
HttpStatusCode code = response.StatusCode;
if (code == HttpStatusCode.OK)
{
using (StreamReader sr = new StreamReader(response.GetResponseStream(), Encoding.UTF8))
{
HtmlDocument htmlDoc = new HtmlDocument();
htmlDoc.OptionFixNestedTags = true;
htmlDoc.Load(sr);
Console.Write(htmlDoc.DocumentNode.InnerText);
}
}
}

C# - How to get cookies from website using xNet

I want to submit Emails in this website: https://www.bitdefender.com/site/Facebook/redFreeDownload?nfo%5Bemail%5D=agytnpbu%40gmx.com&nfo%5BreGoogle%5D=auto-detect&g-recaptcha-response=&nfo%5Bhash_page%5D=tsmd-2016-facebook&nfo%5Bsystem-2016%5D=active
There is a captcha but it doesn't work.
I use Fiddler to get all the information I need. However, I don't know how I can read the cookies so that I will be able to submit emails again and again...
How can I do it? Cookies:
__cfduid=d7132cd01781606b71aa23e43dca589bc1536917322; PHPSESSID=a4ujho0plamjc6nq4n31pqu627; _country=il; AWSALB=xAs9RL/e6nVR17J9qYZnooEQedeMW48ZEgx8no+xyhkQxhCjSsrcnc1l/LpjrfL8vBNXdA40agM6Zk3e4i84DGCDXd/7TqGaaYrb5zeyGHbxy8nBZqIGiKUKtKLV; bd112=3ZAxb4MwEIX%2FiyU6hRAgUIUqilRF6dq9ripjH2AFc5Z9lEZV%2Fnup24GhS9Zup3ffu3d6L59sdD2rWEdkK57wZJqmda1JQQODAreWaHjiNQFPTkJCjXjmiQN1cgBHnIYehToMDUbFIxih%2B6g47kV7ocHWY7TdtObj%2B8SdMPbhh3LwhNj2EMCRMFZAICkQbexACkuyE%2FPkLQ4e9gtvJ3z3ZkUbzOSNirNNWsbN72ML0l88gQnrECRJvwNbMdIGPM0cq9IiL3fpfZ5l19UtNTw7NEh6%2Fm1WrAOhlsrfGfnutox%2FWvW2vL5%2BAQ%3D%3D
My code:
using (var request = new HttpRequest())
{
request.Proxy = HttpProxyClient.Parse("127.0.0.1:8888");
request.IgnoreProtocolErrors = true;
request.KeepAlive = true;
request.ConnectTimeout = 10000;
request.AllowAutoRedirect = false;
request.AddParam("nfo[email]", "kirtchukdaniel#gmail.com");
request.AddParam("nfo[reGoogle]", "auto-detect");
request.AddParam("g-recaptcha-response", "");
request.AddParam("nfo[hash_page]", "tsmd-2016-facebook");
request.AddParam("nfo[system-2016]", "active");
request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.140 Safari/537.36 Edge/17.17134";
request.AddHeader("Host ", "www.bitdefender.com");
request.Referer = "https://www.bitdefender.com/site/Facebook/redFreeDownload?nfo%5Bemail%5D=agytnpbu%40gmx.com&nfo%5BreGoogle%5D=auto-detect&g-recaptcha-response=&nfo%5Bhash_page%5D=tsmd-2016-facebook&nfo%5Bsystem-2016%5D=active";
var attempt = request.Get("https://www.bitdefender.com/site/Promotions/spreadPromotions/").ToString();
Console.WriteLine(attempt);
}
Console.ReadKey();

answer is simple.
request.Cookies = new CookieDictionary();
also remove this(or it can throw an exception).
request.AddHeader("Host ", "www.bitdefender.com");
// =>
// request.AddHeader("Host ", "www.bitdefender.com");

Logging into Wordpress

I am attempting to make a program that automatically downloads some files from a Wordpress site but the problem is you have to be authenticated to download them. I have looked around but can't find anything that will really help me understand this.
This is the code that does the logging in but from what I am gathering it is not storing all the needed cookies for Wordpress, but I haven't found many examples regarding this.
var baseAddress = new Uri("https://www.gplvault.com/wp-admin");
var cookieContainer = new CookieContainer();
using (var handler = new HttpClientHandler() { CookieContainer = cookieContainer, AllowAutoRedirect = false})
using (var client = new HttpClient(handler, false) { BaseAddress = baseAddress })
{
client.DefaultRequestHeaders.Add("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36");
client.DefaultRequestHeaders.Add("Referer", "https://www.gplvault.com/my-account/");
client.DefaultRequestHeaders.Add("Connection", "keep-alive");
client.DefaultRequestHeaders.Add("Accept-Language", "en-US,en;q=0.5");
client.DefaultRequestHeaders.Add("Accept", "application/json, text/javascript, */*; q=0.01");
var homePageResult = client.GetAsync("/");
Console.WriteLine("1: " + cookieContainer.GetCookieHeader(baseAddress));
homePageResult.Result.EnsureSuccessStatusCode();
var content = new FormUrlEncodedContent(new[]{
new KeyValuePair<string, string>("username", username),
new KeyValuePair<string, string>("password", password),
});
var loginResult = client.PostAsync("/my-account/orders/", content).Result;
}
I used tamper to get the headers and this is what I got:
wget
--header="Host: www.gplvault.com"
--header="User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"
--header="Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8"
--header="Accept-Language: en-US,en;q=0.9"
--header="Cookie: __cfduid=dad983khm234b5jh23s39421e59ce2d1534278105; _ga=GA1.2.1355145636.1534278108; _gid=GA1.2.858287631.1577566529; wordpress_logged_in_13cb68234k56h43k56hk34g22133fe5=some.user%7C1535739391%7CZMz24UN1y2XPWpa1b22yfDi456fdgDGFdfg5jFufjlyKr54%7C68b0b014c9ba2dba566aa0beb44a99f60f97869dFGdfs11125361b3a2e908cd5; wfwaf-authcookie-f227c3e58kjhaf97s98dfasd9081b332=789%7Csubscriber%7Cedbc35748c970a9d4a441209cfc01e5283d791968a1cc28aa740424c08bca223; woocommerce_items_in_cart=1; woocommerce_cart_hash=faece8037500a234545305461c8ca026; wp_woocommerce_session_13cb682d9769DFGsdsb449b7722133fe5=789%7C%7C1535739438%7C%7C1535735838%7C%7C7e56398513f91b876s890d4d2f8ccff3b; _gat=1"
--header="Connection: keep-alive" "https://www.gplvault.com/download/21377/" -O "codecanyon-9899552-actionable-google-analytics-for-woocommerce.zip" -c

Not able to download file using webclient, but I am able to download it manually from the website

private void download_nse()
{
string source_location = "https://www.nseindia.com/content/historical/DERIVATIVES/2015/DEC/fo23DEC2015bhav.csv.zip";
Uri uu = new Uri(source_location);
using (WebClient fileReader = new WebClient())
{
try
{
var ua = "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36";
fileReader.Headers.Add(HttpRequestHeader.UserAgent, ua);
fileReader.Headers["Accept"] = "/";
fileReader.DownloadFile(uu, #"d:\nse\notworking.zip");
fileReader.Dispose();
}
catch
{
}
}
source_location = "https://www.nseindia.com/content/historical/DERIVATIVES/2016/JAN/fo05JAN2016bhav.csv.zip";
using (WebClient fileReader = new WebClient())
{
try
{
var ua = "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36";
fileReader.Headers.Add(HttpRequestHeader.UserAgent, ua);
fileReader.Headers["Accept"] = "/";
fileReader.DownloadFile(source_location, #"d:\nse\working.zip");
}
catch
{
}
}
}

I've checked the url you gave and see that when you select the "Date(DD-MM-YYYY)" and "Select Report" drop-down options and click on "Get Data", there is a GET request send to https://www.nseindia.com/ArchieveSearch with three parameter like this: "?h_filetype=fobhav&date=02-04-2018&section=FO". And this GET request returns:
<p class="archive_title">F&O - Bhavcopy for 02-04-2018 </p><br>
<br><table cellpadding=5>
<tr>
<td class=t0><a href=/content/historical/DERIVATIVES/2018/APR/fo02APR2018bhav.csv.zip target=new>fo02APR2018bhav.csv.zip</a></td></tr>
<tr>
</table>
</td></tr>
</table>
which contains the link to download file. so when you send the POST request without the GET it fails.
You can send a GET request first and then send POST with the returned link like this:
string source_location2 = "https://www.nseindia.com/ArchieveSearch";
Uri uu2 = new Uri(source_location2);
using (WebClient fileReader = new WebClient())
{
try
{
var ua = "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36";
fileReader.Headers.Add(HttpRequestHeader.UserAgent, ua);
fileReader.Headers["Accept"] = "/";
fileReader.QueryString.Add("h_filetype", "fobhav");
fileReader.QueryString.Add("date", "02-04-2018");
fileReader.QueryString.Add("section", "FO");
var response = fileReader.DownloadString(uu2);
//using Html Agility Pack to parse the response and get the download link. you need to add this package through nuget package manager to project for this code to work.
var htmlDoc = new HtmlDocument();
htmlDoc.LoadHtml(response);
var fileLink = htmlDoc.DocumentNode.SelectSingleNode("//table//tr//td//a").Attributes["href"].Value;
//now sending the actual file request...
var fileReader2 = new WebClient();
fileReader2.Headers.Add(HttpRequestHeader.UserAgent, ua);
fileReader2.Headers["Accept"] = "/";
fileReader2.DownloadFile(new Uri("https://www.nseindia.com" + fileLink), #"d:\notworking.zip");
fileReader2.Dispose();
}
catch(Exception e)
{
throw e;
}
}