I spent several hours debuging the code, playing with Fiddler and googling, but still no luck, so hopefully you will help me.
I am trying to get the source of http://www.finishline.com. The catch is, the HttpWebRequest works in some regions (like here in Slovakia), but doesn't work in USA what I need to achieve.
For USA the request.GetResponse() just time outs. I have tried countless headers combinations, but without success. Can you please help? Thank you
var request = (HttpWebRequest)WebRequest.Create("http://www.finishline.com");
request.CookieContainer = new CookieContainer();
request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36";
request.AutomaticDecompression = (DecompressionMethods.GZip | DecompressionMethods.Deflate);
request.Headers.Add("Accept-Encoding", "gzip, deflate");
request.Accept = " text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
request.Headers.Add("Upgrade-Insecure-Requests", "1");
request.Headers.Add("Accept-Language", "sk,cs;q=0.8,en-US;q=0.5,en;q=0.3");
request.KeepAlive = true;
request.Headers.Add("Cache-Control", "max-age=0");
var responseText = "";
using (var response = request.GetResponse())
{
var httpWebResponse = response.GetResponseStream();
using (var sr = new StreamReader(httpWebResponse))
{
responseText = sr.ReadToEnd();
}
}
There are two kind of timeouts. Client timeout and server timeout. Have you tried doing something like this:
request.Timeout = Timeout.Infinite;
request.KeepAlive = true;
So, your GetResponse never get timedout you can check with it.
Or
using (HttpWebResponse response = (HttpWebResponse) request.GetResponse())
{
var httpWebResponse = response.GetResponseStream();
using (var sr = new StreamReader(httpWebResponse))
{
responseText = sr.ReadToEnd();
}
}
Try this method to fix your issue.
Related
I am trying to call my API reference but when it gets to the HttpWebResponse response = (HttpWebResponse)request.GetResponse()) section my code just returns the error 500. I'm unsure on what my code is actually missing as im pretty new to APIs
any help is appreciated.
var request = (HttpWebRequest)WebRequest.Create($"####");
request.Method = "GET";
request.ContentType = "application/json";
request.Headers.Add($"USERNAME: {Username}, PASSWORD: {Password}");
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.75 Safari/537.36";
request.ServicePoint.Expect100Continue = false;
request.ProtocolVersion = HttpVersion.Version11;
string responseFromServer = "";
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
//Console.WriteLine(((HttpWebResponse)response).StatusDescription);
using (var dataStream = response.GetResponseStream())
{
StreamReader reader = new StreamReader(dataStream);
responseFromServer = reader.ReadToEnd();
//Console.WriteLine(responseFromServer);
}
A 500 HTTP response code is a server based error, your code is not the issue. However, some "poorly made" APIs do not handle insufficient keys with the proper HTTP codes, resulting in a server error due to the front end not sending the required keys/values in the request.
Good luck!
My application is downloading many web pages and files and I don't want it to ever spend more than 4 seconds trying to download a webpage or file. I can't get the correct timeout settings to allow this.
What setting an I missing here?
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(websiteURL);
request.Timeout = 4000;
request.KeepAlive = false;
request.AllowAutoRedirect = true;
request.ProtocolVersion = HttpVersion.Version11;
request.Accept = "text/html, application/xhtml+xml, application/xml; q=0.9, image/webp, */*; q=0.8";
request.Headers["Accept-Language"] = "en-GB,en-US;q=0.7,en;q=0.3";
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.97 Safari/537.36";
request.ReadWriteTimeout = 4000;
request.AllowWriteStreamBuffering = false;
response = (HttpWebResponse)request.GetResponse();
Stream dataStream = response.GetResponseStream();
StreamReader reader = new StreamReader(dataStream);
reader.BaseStream.ReadTimeout = 4000;
dataStream.WriteTimeout = 4000;
dataStream.ReadTimeout = 4000;
responseFromServer = reader.ReadToEnd();
You should use httpClient instead of HttpWebRequest.
I used httpClient and timeout worked fine.
Try this:
using (var httpClient = new HttpClient())
{
httpClient.Timeout = new TimeSpan(0, 0, 4);
Stream response = await httpClient.GetStreamAsync("https://www.website.com/");
StreamReader reader = new StreamReader(response);
}
HttpClient vs HttpWebRequest
Trying to POST a webrequest but not getting the actual results.. webrequest not getting submit. Previously this code was working fine in 2018. I'm not sure if they changed something at their end. I copied all the latest parameters, and other stuff from fiddler. Is there anyone please look into it as my deadline for the actual project is very tight and this code is just an integral part of the main project and was assuming it is functional already.. already spent few hours on this..
CookieContainer _cookies = new CookieContainer();
string urls = "https://ww3.freddiemac.com/loanlookup?Phone=&Email=&Contact=&lang=en&fname=TEST&lname=PERSON&strt_nbr=123&unit_num=&strt_name=FAKE STREET&strt_suffix=&addr_city=CHICAGO&addr_state=IL&addr_zip=60628&ssn=1234&Verify=Yes";
var request = (HttpWebRequest)WebRequest.Create(urls);
request.Host = "ww3.freddiemac.com";
request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36 Edg/83.0.478.61";
request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9";
WebHeaderCollection myWebHeaderCollection = request.Headers;
myWebHeaderCollection.Add("Accept-Language", "en-US,en;q=0.9");
request.Headers["Accept-Encoding"] = "gzip, deflate, br";
request.ContentType = "application/x-www-form-urlencoded";
request.Method = "POST";
request.Referer = "https://ww3.freddiemac.com/loanlookup";
var encoding = new ASCIIEncoding();
var postBytes = encoding.GetBytes(urls);
request.AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip | DecompressionMethods.None;
request.ContentLength = postBytes.Length;
request.CookieContainer = _cookies;
request.KeepAlive = true;
var requestStream = request.GetRequestStream();
requestStream.Write(postBytes, 0, postBytes.Length);
requestStream.Close();
var response = (HttpWebResponse)request.GetResponse();
string html;
using (var reader = new StreamReader(response.GetResponseStream()))
{
html = reader.ReadToEnd();
}
if (html.Contains("No. Our records show"))
{
_qualified = false;
}
At the time of opening, using HttpWebRequest. Why are visiting
https://web4.sa8888.net/sport/Games.aspx?lang=3&device=pc
Can't get the data inside?
string strResult = "";
try
{
string url = "https://web4.sa8888.net/sport/Games.aspx?lang=3&device=pc";
ServicePointManager.Expect100Continue = true;
ServicePointManager.SecurityProtocol = System.Net.SecurityProtocolType.Tls12;
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.AutomaticDecompression = DecompressionMethods.GZip;
request.Method = "GET";
request.Timeout = 30000;
request.KeepAlive = true;
request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36";
request.ContentType = "application/x-www-form-urlencoded; charset=UTF-8";
request.Headers.Set("Pragma", "no-cache");
//request.ContentLength = bs.Length;
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Stream streamReceive = response.GetResponseStream();
Encoding encoding = Encoding.GetEncoding("UTF-8");
StreamReader streamReader = new StreamReader(streamReceive, encoding);
strResult = streamReader.ReadToEnd();
}
catch (Exception e)
{
MessageBox.Show(e.Message);
}
textBox1.Text=strResult;
The page you've linked to is built using javascript so what you are seeing is not raelly part of the source. If you look at for example https://web4.sa8888.net/js/v1000/sport/games/IceHockey.js you will see that they build alot of the table data there.
HttpWebRequest will only fetch a file for you, it won't run javascript and such like a real browser would. Thats why you are not getting the data you want.
After looking at it a bit more it seems like they are streaming information in a binary format via websockets so this is a very non-trivial setup to get the data out (and probably at least partially built to avoid people scraping their data).
First thing, the site and post.
https://saplic.receita.pb.gov.br/sintegra/SINf_ConsultaSintegra.jsp
Fullfil Field CNPJ with value For instance 34151100004209.
Works fine in Chrome and .Net 4.0 Httpwebrequest. But I cant debug in fiddler and cant make it work on .Net 3.5
I'm already using
System.Net.ServicePointManager.ServerCertificateValidationCallback += new System.Net.Security.RemoteCertificateValidationCallback(customXertificateValidation);
System.Net.ServicePointManager.SecurityProtocol = System.Net.SecurityProtocolType.Ssl3;
System.Net.ServicePointManager.Expect100Continue = false;
This is driving me insane. Any help will be greatly appreciated
To make things more clear, i'm having one problem that is happening on 2 differente places (Fiddler and .net 3.5)
The code i'm trying to run is.
CookieContainer CookCon = new CookieContainer();
HttpWebRequest Request = (HttpWebRequest)WebRequest.Create("http://sintegra.receita.pb.gov.br/sintegra/sintegra.asp?estado=pb");
Request.Timeout = Configuration.TimeOut;
Request.Proxy = Configuration.Proxy;
Request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.116 Safari/537.36";
Request.Headers.Add("Accept-Encoding", "gzip, deflate");
Request.CookieContainer = new CookieContainer();
using (HttpWebResponse Response = Request.GetResponse() as HttpWebResponse)
{
foreach (Cookie C in Response.Cookies)
{
CookCon.Add(C);
}
Response.Close();
}
HttpWebRequest Request = (HttpWebRequest)WebRequest.Create("https://saplic.receita.pb.gov.br/sintegra/");
Request.Referer = "http://sintegra.receita.pb.gov.br/sintegra/sintegra.asp?estado=pb";
Request.Headers.Add("X-DevTools-Emulate-Network-Conditions-Client-Id", "C5E5561B-11F5-4D9E-A9D4-54A18E660D11");
Request.CookieContainer = CookCon;
Request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8";
Request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.103 Safari/537.36";
string HTML = "";
string Encoding = "ISO-8859-1";
Request.Timeout = 20000;
Request.Headers.Add("Accept-Encoding", "gzip, deflate");
using (HttpWebResponse Response = Request.GetResponse() as HttpWebResponse)
{
Stream S = Response.GetResponseStream();
if (Response.ContentEncoding.ToLower().Contains("gzip"))
{
S = new GZipStream(S, CompressionMode.Decompress);
using (System.IO.StreamReader ResStream = new StreamReader(S, System.Text.Encoding.GetEncoding(Encoding)))
{
HTML = ResStream.ReadToEnd();
ResStream.Close();
}
}
else if (Response.ContentEncoding.ToLower().Contains("deflate"))
{
S = new DeflateStream(S, CompressionMode.Decompress);
using (System.IO.StreamReader ResStream = new StreamReader(S, System.Text.Encoding.GetEncoding(Encoding)))
{
HTML = ResStream.ReadToEnd();
ResStream.Close();
}
}
else
{
using (System.IO.StreamReader ResStream = new StreamReader(Response.GetResponseStream(), System.Text.Encoding.GetEncoding(Encoding)))
{
HTML = ResStream.ReadToEnd();
ResStream.Close();
}
}
S.Close();
S.Dispose();
Response.Close();
}
The error i'm getting is the "unexpeted EOF" excpetion when i try to make the post(GetResponse part of the code above).
If i try to surf the site on Chrome with Fiddler on, it wont return northing, just keeps on hold, hold, hold and nothing.
If I copy the same code to a new 4.0 project, it runs without problem.
If its not clear, please post the doubts, I'm trying to be as clear as possible.
Since no one helped, i just made a webservice using the .net 4.0 framework and I'm calling it from the 3.5 app.
Still no clue why this happens..