I'm trying to get simple gzip encoded html response from a website and it keeps getting time out, following is my code:
HttpWebRequest httpClient = (HttpWebRequest)WebRequest.Create(url);
httpClient.Method = "GET";
httpClient.Accept = "text/html, application/xhtml+xml, */*";
httpClient.Headers.Add("Accept-Encoding: gzip, deflate");
httpClient.Headers.Add("Accept-Language: en-US");
httpClient.Headers.Add("DNT: 1");
httpClient.ProtocolVersion = HttpVersion.Version10;
httpClient.KeepAlive = true;
httpClient.Timeout = System.Threading.Timeout.Infinite;
httpClient.CookieContainer = cookieJar;
String responseAsText;
using (HttpWebResponse response = (HttpWebResponse)httpClient.GetResponse())
{
System.IO.StreamReader sr;
if (response.ContentEncoding.Equals("gzip"))
{
sr = new StreamReader(new GZipStream(response.GetResponseStream(), CompressionMode.Decompress));
}
else
{
sr = new System.IO.StreamReader(response.GetResponseStream());
}
responseAsText = sr.ReadToEnd();
}
The url I'm trying to hit is "https client.schwab.com/Login/SignOn/CustomerCenterLogin.aspx"
This works perfectly fine in the Browser, using Fiddler I viewed the browser's Request header and since its Transfer-Encoding: chunked, I have used HttpVersion10
I have also tried setting httpClient.Timeout = System.Threading.Timeout.Infinite, but it never gets back with a response, however in browser the response gets in few seconds.
Please someone help me in achieving this.
probably you can try setting Agent property, so it doesn't recognize you as a bot.
I think Nero has answered your question ..
Try adding these Lines in your code..
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101 Firefox/24.0";
Related
I'm developing an app for a school project which uses data from CoinGecko's free public API. The way I'm getting the data is with the HttpWebRequest class. At first, I was able to use the API fast, without any problems, but recently the requests have been getting stuck at the GetResponse() function for a long time. Even pinging the API takes about 30 seconds to complete. I've read a lot of posts regarding this problem, and none of the suggestions worked.
try
{
string uri = "https://api.coingecko.com/api/v3/ping";
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uri);
request.Method = "GET";
request.Accept =
"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng," +
"*/*;q=0.8,application/signed-exchange;v=b3;q=0.9";
request.ContentType = "application/json";
request.Headers.Add("accept-encoding", "gzip, deflate, br");
request.Headers.Add("accept-language", "en-US,en;q=0.9");
request.Headers.Add("cache-control", "max-age=0");
request.Headers.Add("if-none-match", "W/\"c1f074e1bdf979ac5dc291f2c0acf9e4\"");
request.Headers.Add("sec-ch-ua", "\" Not A;Brand\"; v = \"99\", \"Chromium\"; v = \"96\", \"Google Chrome\"; v = \"96\"");
request.Headers.Add("sec-ch-ua-mobile", "?0");
request.Headers.Add("sec-ch-ua-platform", "\"Windows\"");
request.Headers.Add("sec-fetch-dest", "document");
request.Headers.Add("sec-fetch-mode", "navigate");
request.Headers.Add("sec-fetch-site", "none");
request.Headers.Add("sec-fetch-user", "?1");
request.Headers.Add("upgrade-insecure-requests", "1");
request.UserAgent =
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko)" +
" Chrome/97.0.4692.71 Safari/537.36 Edg/97.0.1072.55";
request.Proxy = null;
using HttpWebResponse response = (HttpWebResponse)request.GetResponse();
if (response.StatusCode != HttpStatusCode.OK && response.StatusCode != HttpStatusCode.Created)
{
throw new ApplicationException("REST client error code: "
+ response.StatusCode.ToString());
}
using Stream stream = response.GetResponseStream();
if (stream == null)
{
throw new ApplicationException("Error: Null response");
}
using StreamReader reader = new StreamReader(stream);
return reader.ReadToEnd();
}
catch (Exception ex)
{
return ex.ToString();
}
It works with the browser so I copied all the browser request headers and added them to the code but they didn't help. I have also tried: Increasing the number of connections, playing around with all the security protocol types, disabling my firewall. Could it be that the API is detecting and slowing down my requests since they're not "human"?
Set Windows to prioritise IPv4 by typing the following in the command line:
netsh interface ipv6 set prefixpolicy ::ffff:0:0/96 46 4
Source
I spent several hours debuging the code, playing with Fiddler and googling, but still no luck, so hopefully you will help me.
I am trying to get the source of http://www.finishline.com. The catch is, the HttpWebRequest works in some regions (like here in Slovakia), but doesn't work in USA what I need to achieve.
For USA the request.GetResponse() just time outs. I have tried countless headers combinations, but without success. Can you please help? Thank you
var request = (HttpWebRequest)WebRequest.Create("http://www.finishline.com");
request.CookieContainer = new CookieContainer();
request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36";
request.AutomaticDecompression = (DecompressionMethods.GZip | DecompressionMethods.Deflate);
request.Headers.Add("Accept-Encoding", "gzip, deflate");
request.Accept = " text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
request.Headers.Add("Upgrade-Insecure-Requests", "1");
request.Headers.Add("Accept-Language", "sk,cs;q=0.8,en-US;q=0.5,en;q=0.3");
request.KeepAlive = true;
request.Headers.Add("Cache-Control", "max-age=0");
var responseText = "";
using (var response = request.GetResponse())
{
var httpWebResponse = response.GetResponseStream();
using (var sr = new StreamReader(httpWebResponse))
{
responseText = sr.ReadToEnd();
}
}
There are two kind of timeouts. Client timeout and server timeout. Have you tried doing something like this:
request.Timeout = Timeout.Infinite;
request.KeepAlive = true;
So, your GetResponse never get timedout you can check with it.
Or
using (HttpWebResponse response = (HttpWebResponse) request.GetResponse())
{
var httpWebResponse = response.GetResponseStream();
using (var sr = new StreamReader(httpWebResponse))
{
responseText = sr.ReadToEnd();
}
}
Try this method to fix your issue.
i'am using burp suit to check the requests and i m trying to convertthis to c# code
POST /sso HTTP/1.1
Host: account.ankama.com
User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:51.0) Gecko/20100101 Firefox/51.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: fr,fr-FR;q=0.8,en-US;q=0.5,en;q=0.3
Referer: http://www.dofus.com/fr
Cookie: LANG=fr; _ga=GA1.1.1197518596.1489526959; SID=452EDCF3C4BD32057F9F08254BE40001
Connection: close
Upgrade-Insecure-Requests: 1
Content-Type: application/x-www-form-urlencoded
Content-Length: 102
action=login&from=http%3A%2F%2Fwww.dofus.com%2Ffr&login=user123&password=password1232F&remember=1
So i tried to :
HttpWebRequest Request = (HttpWebRequest)WebRequest.Create("https://account.ankama.com/sso?action=login&from=https%3A%2F%2Faccount.ankama.com%2Ffr%2Fsecurite%2Fmode-restreint%3Ff%3Dhttps%3A%2F%2Faccount.ankama.com%2Ffr%2Fidentification%3Ff%3Dhttps%3A%2F%2Faccount.ankama.com%2Ffr%2Fcompte%2Finformations&login=user111&password=password1472F");
Request.ContentType = "application/x-www-form-urlencoded";
Request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
Request.UserAgent = "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:51.0) Gecko/20100101 Firefox/51.0";
Request.Host = "account.ankama.com";
Request.Referer = "https://account.ankama.com/fr/votre-compte/profil";
Request.Method = "POST";
Request.AllowAutoRedirect = true;
Request.CookieContainer = new CookieContainer();
//quest.Credentials = new NetworkCredential("user123", "passowrd123");
using (HttpWebResponse response = (HttpWebResponse)Request.GetResponse())
{
using (Stream stream = response.GetResponseStream())
{
StreamReader reader = new StreamReader(stream);
StreamWriter writer = new StreamWriter("odm.html");
writer.Write(reader.ReadToEnd());
writer.Close();
reader.Close();
Console.WriteLine("Done");
}
}
Console.ReadKey();
in the file odm.html I m checking if the html code contain "My account" that shown when the user is actually logged in .
but this doesnt seems to be working for some reasons that i still don't know .
i made some research to about HTTP status code but in my brup suit after trying to login in with an actual exisiting account and a none valid account it gives the same http code 302 with a different Content Length .
EDIT:
the issue is i don't find 'my account' in the html file , i only find the page where the user is going to login
You are trying to send request body in query string, you are setting the request method as POST but you are not sending the body. The request url should be:
https://account.ankama.com/sso
And you need to set request body before sending the request:
var bytes = Encoding.UTF8.GetBytes("action=login&from=http%3A%2F%2Fwww.dofus.com%2Ffr&login=user123&password=password1232F&remember=1");
request.ContentLength = bytes.Length;
using (var stream = request.GetRequestStream())
{
stream.Write(bytes, 0, bytes.Length);
}
First thing, the site and post.
https://saplic.receita.pb.gov.br/sintegra/SINf_ConsultaSintegra.jsp
Fullfil Field CNPJ with value For instance 34151100004209.
Works fine in Chrome and .Net 4.0 Httpwebrequest. But I cant debug in fiddler and cant make it work on .Net 3.5
I'm already using
System.Net.ServicePointManager.ServerCertificateValidationCallback += new System.Net.Security.RemoteCertificateValidationCallback(customXertificateValidation);
System.Net.ServicePointManager.SecurityProtocol = System.Net.SecurityProtocolType.Ssl3;
System.Net.ServicePointManager.Expect100Continue = false;
This is driving me insane. Any help will be greatly appreciated
To make things more clear, i'm having one problem that is happening on 2 differente places (Fiddler and .net 3.5)
The code i'm trying to run is.
CookieContainer CookCon = new CookieContainer();
HttpWebRequest Request = (HttpWebRequest)WebRequest.Create("http://sintegra.receita.pb.gov.br/sintegra/sintegra.asp?estado=pb");
Request.Timeout = Configuration.TimeOut;
Request.Proxy = Configuration.Proxy;
Request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.116 Safari/537.36";
Request.Headers.Add("Accept-Encoding", "gzip, deflate");
Request.CookieContainer = new CookieContainer();
using (HttpWebResponse Response = Request.GetResponse() as HttpWebResponse)
{
foreach (Cookie C in Response.Cookies)
{
CookCon.Add(C);
}
Response.Close();
}
HttpWebRequest Request = (HttpWebRequest)WebRequest.Create("https://saplic.receita.pb.gov.br/sintegra/");
Request.Referer = "http://sintegra.receita.pb.gov.br/sintegra/sintegra.asp?estado=pb";
Request.Headers.Add("X-DevTools-Emulate-Network-Conditions-Client-Id", "C5E5561B-11F5-4D9E-A9D4-54A18E660D11");
Request.CookieContainer = CookCon;
Request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8";
Request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.103 Safari/537.36";
string HTML = "";
string Encoding = "ISO-8859-1";
Request.Timeout = 20000;
Request.Headers.Add("Accept-Encoding", "gzip, deflate");
using (HttpWebResponse Response = Request.GetResponse() as HttpWebResponse)
{
Stream S = Response.GetResponseStream();
if (Response.ContentEncoding.ToLower().Contains("gzip"))
{
S = new GZipStream(S, CompressionMode.Decompress);
using (System.IO.StreamReader ResStream = new StreamReader(S, System.Text.Encoding.GetEncoding(Encoding)))
{
HTML = ResStream.ReadToEnd();
ResStream.Close();
}
}
else if (Response.ContentEncoding.ToLower().Contains("deflate"))
{
S = new DeflateStream(S, CompressionMode.Decompress);
using (System.IO.StreamReader ResStream = new StreamReader(S, System.Text.Encoding.GetEncoding(Encoding)))
{
HTML = ResStream.ReadToEnd();
ResStream.Close();
}
}
else
{
using (System.IO.StreamReader ResStream = new StreamReader(Response.GetResponseStream(), System.Text.Encoding.GetEncoding(Encoding)))
{
HTML = ResStream.ReadToEnd();
ResStream.Close();
}
}
S.Close();
S.Dispose();
Response.Close();
}
The error i'm getting is the "unexpeted EOF" excpetion when i try to make the post(GetResponse part of the code above).
If i try to surf the site on Chrome with Fiddler on, it wont return northing, just keeps on hold, hold, hold and nothing.
If I copy the same code to a new 4.0 project, it runs without problem.
If its not clear, please post the doubts, I'm trying to be as clear as possible.
Since no one helped, i just made a webservice using the .net 4.0 framework and I'm calling it from the 3.5 app.
Still no clue why this happens..
I am working with HTTPWebRequests for the first time now and i'm trying to wrap my head around it all. I am trying to populate the "Start Date" text box on this website http://treasurer.maricopa.gov/Parcel/TaxReceipt.aspx. I used Fiddler to find the incredibly long Json query that comes back from the POST. ( i will post the entire thing if needed )...but i noticed the last part is what I'm after
"............,"StartDate":"1/1/2013","EndDate":"12/31/2013"}}
So far, this is the function i am using but the result comes back without any changes (so it's as if you just went to the website i first posted)
public static void json(string url)
{
var httpWebRequest = (HttpWebRequest)WebRequest.Create(url);
httpWebRequest.ContentType = "text/json";
httpWebRequest.Method = "POST";
httpWebRequest.CookieContainer = cookieJar;
//httpWebRequest.Accept = "text/html, application/xhtml+xml, */*";
httpWebRequest.Headers.Add("Accept-Language: en-US");
httpWebRequest.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW65; Trident/7.0; MAM5; rv:11.0) like Gecko";
httpWebRequest.Headers.Add("Accept-Encoding: gzip, deflate");
//httpWebRequest.Referer = url4;
using (var streamWriter = new StreamWriter(httpWebRequest.GetRequestStream()))
{
string json = #"{StartDate: 1/1/2013}, {EndDate: 12/13/2013}";
streamWriter.Write(json);
streamWriter.Flush();
streamWriter.Close();
var httpResponse = (HttpWebResponse)httpWebRequest.GetResponse();
using (var streamReader = new StreamReader(httpResponse.GetResponseStream()))
{
var result = streamReader.ReadToEnd();
}
}