My application is downloading many web pages and files and I don't want it to ever spend more than 4 seconds trying to download a webpage or file. I can't get the correct timeout settings to allow this.
What setting an I missing here?
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(websiteURL);
request.Timeout = 4000;
request.KeepAlive = false;
request.AllowAutoRedirect = true;
request.ProtocolVersion = HttpVersion.Version11;
request.Accept = "text/html, application/xhtml+xml, application/xml; q=0.9, image/webp, */*; q=0.8";
request.Headers["Accept-Language"] = "en-GB,en-US;q=0.7,en;q=0.3";
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.97 Safari/537.36";
request.ReadWriteTimeout = 4000;
request.AllowWriteStreamBuffering = false;
response = (HttpWebResponse)request.GetResponse();
Stream dataStream = response.GetResponseStream();
StreamReader reader = new StreamReader(dataStream);
reader.BaseStream.ReadTimeout = 4000;
dataStream.WriteTimeout = 4000;
dataStream.ReadTimeout = 4000;
responseFromServer = reader.ReadToEnd();
You should use httpClient instead of HttpWebRequest.
I used httpClient and timeout worked fine.
Try this:
using (var httpClient = new HttpClient())
{
httpClient.Timeout = new TimeSpan(0, 0, 4);
Stream response = await httpClient.GetStreamAsync("https://www.website.com/");
StreamReader reader = new StreamReader(response);
}
HttpClient vs HttpWebRequest
Related
Trying to POST a webrequest but not getting the actual results.. webrequest not getting submit. Previously this code was working fine in 2018. I'm not sure if they changed something at their end. I copied all the latest parameters, and other stuff from fiddler. Is there anyone please look into it as my deadline for the actual project is very tight and this code is just an integral part of the main project and was assuming it is functional already.. already spent few hours on this..
CookieContainer _cookies = new CookieContainer();
string urls = "https://ww3.freddiemac.com/loanlookup?Phone=&Email=&Contact=&lang=en&fname=TEST&lname=PERSON&strt_nbr=123&unit_num=&strt_name=FAKE STREET&strt_suffix=&addr_city=CHICAGO&addr_state=IL&addr_zip=60628&ssn=1234&Verify=Yes";
var request = (HttpWebRequest)WebRequest.Create(urls);
request.Host = "ww3.freddiemac.com";
request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36 Edg/83.0.478.61";
request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9";
WebHeaderCollection myWebHeaderCollection = request.Headers;
myWebHeaderCollection.Add("Accept-Language", "en-US,en;q=0.9");
request.Headers["Accept-Encoding"] = "gzip, deflate, br";
request.ContentType = "application/x-www-form-urlencoded";
request.Method = "POST";
request.Referer = "https://ww3.freddiemac.com/loanlookup";
var encoding = new ASCIIEncoding();
var postBytes = encoding.GetBytes(urls);
request.AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip | DecompressionMethods.None;
request.ContentLength = postBytes.Length;
request.CookieContainer = _cookies;
request.KeepAlive = true;
var requestStream = request.GetRequestStream();
requestStream.Write(postBytes, 0, postBytes.Length);
requestStream.Close();
var response = (HttpWebResponse)request.GetResponse();
string html;
using (var reader = new StreamReader(response.GetResponseStream()))
{
html = reader.ReadToEnd();
}
if (html.Contains("No. Our records show"))
{
_qualified = false;
}
I spent several hours debuging the code, playing with Fiddler and googling, but still no luck, so hopefully you will help me.
I am trying to get the source of http://www.finishline.com. The catch is, the HttpWebRequest works in some regions (like here in Slovakia), but doesn't work in USA what I need to achieve.
For USA the request.GetResponse() just time outs. I have tried countless headers combinations, but without success. Can you please help? Thank you
var request = (HttpWebRequest)WebRequest.Create("http://www.finishline.com");
request.CookieContainer = new CookieContainer();
request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36";
request.AutomaticDecompression = (DecompressionMethods.GZip | DecompressionMethods.Deflate);
request.Headers.Add("Accept-Encoding", "gzip, deflate");
request.Accept = " text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
request.Headers.Add("Upgrade-Insecure-Requests", "1");
request.Headers.Add("Accept-Language", "sk,cs;q=0.8,en-US;q=0.5,en;q=0.3");
request.KeepAlive = true;
request.Headers.Add("Cache-Control", "max-age=0");
var responseText = "";
using (var response = request.GetResponse())
{
var httpWebResponse = response.GetResponseStream();
using (var sr = new StreamReader(httpWebResponse))
{
responseText = sr.ReadToEnd();
}
}
There are two kind of timeouts. Client timeout and server timeout. Have you tried doing something like this:
request.Timeout = Timeout.Infinite;
request.KeepAlive = true;
So, your GetResponse never get timedout you can check with it.
Or
using (HttpWebResponse response = (HttpWebResponse) request.GetResponse())
{
var httpWebResponse = response.GetResponseStream();
using (var sr = new StreamReader(httpWebResponse))
{
responseText = sr.ReadToEnd();
}
}
Try this method to fix your issue.
I've googled and searched here. Some suggest that streams were not being close, others suggested that it's a connection limit with ServicePointManager.DefaultConnectionLimit being set to 1. However, none of these seem to work.
My problem is, when i use this for the first time, it works:
using (var stream = request.GetRequestStream())
{
var data = Encoding.UTF8.GetBytes(post.ToString());
stream.Write(data, 0, data.Length);
}
When I use it a second time, it freezes. Yes, I'm disposing my stream, yes im Aborting and closing my response and requests.
Here is my entire code segment:
public string get_hash(string strUsername, string strPassword, string strUniverse)
{
// Request VAR
var request = (HttpWebRequest)WebRequest.Create("http://website.com/");
// Response VAR
var response = (HttpWebResponse)request.GetResponse();
// Cookie Var
var cookie = new CookieContainer();
ServicePointManager.DefaultConnectionLimit = 100;
request.Timeout = 10;
request = (HttpWebRequest)WebRequest.Create("http://website.com/main/login");
request.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8";
request.Headers.Add("Accept-Encoding", "gzip,deflate,sdch");
request.Headers.Add("Accept-Language", "en-US,en,q=0.8");
request.Headers.Add("Cache-Control", "max-age=0");
request.ContentType = "application/x-www-form-urlencoded";
request.Host = "website.com";
request.Headers.Add("Origin", "http://website.com");
request.Referer = "http://website.com/";
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.117 Safari/537.36";
request.Method = "POST";
request.CookieContainer = new CookieContainer();
request.CookieContainer.Add(cookie.GetCookies(request.RequestUri));
// SET POST DATA HERE
var post = HttpUtility.ParseQueryString(string.Empty);
post.Add("uni", strUni);
post.Add("login", strUsername);
post.Add("pass", strPassword);
using (var stream = request.GetRequestStream())
{
var data = Encoding.UTF8.GetBytes(post.ToString());
stream.Write(data, 0, data.Length);
stream.Close();
stream.Dispose();
}
response = (HttpWebResponse)request.GetResponse();
string strSSID = "Failed";
if (response.StatusCode == HttpStatusCode.OK)
{
var data = string.Empty;
using (var sReader = new StreamReader(response.GetResponseStream()))
{
data = sReader.ReadToEnd();
sReader.Close();
sReader.Dispose();
}
string strSSIDurl = response.ResponseUri.ToString();
int intSSIDurlStart = strSSIDurl.IndexOf("PHPSESSID=") + 10;
strSSID = strSSIDurl.Substring(intSSIDurlStart);
}
request.Abort();
response.Close();
response.Dispose();
return strSSID;
}
you are not disposing the response from the first request.
var request = (HttpWebRequest)WebRequest.Create("http://website.com/");
var response = (HttpWebResponse)request.GetResponse(); // Not disposed
We had similar problem and disposing the response properly helped us.
This example login function But does not work for loading the Dashboard page wordpress.
I need to perform some action in wordpress admin panel programmatically but can't manage how to login to Wordpress using C# and HttpWebRequest.
Here is what I do:
namespace Wordpress_Login
{
public class WPWorkerTests
{
CookieContainer cookieJar = new CookieContainer();
public String StartAdmin()
{
//string wp_url = "http://localhost/wp";
string admin_url = "http://localhost/wp/wp-admin";
string login_url = "http://localhost/wp/wp-login.php";
string post_data = "log=test&pwd=asdfz&rememberme=forever&wp-submit=Log+In&redirect_to=http%3A%2F%2Flocalhost%2Fwp%2Fwp-admin%2F&testcookie=1";
HttpWebRequest request1 = (HttpWebRequest)WebRequest.Create(login_url);
request1.CookieContainer = cookieJar;
request1.UserAgent = "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:25.0) Gecko/20100101 Firefox/25.0";
request1.KeepAlive = true; //this is the default
request1.ContentType = "text/html";
request1.Timeout = 10000;
request1.Method = "GET";
HttpWebResponse response1 = (HttpWebResponse)request1.GetResponse();
StreamReader sr = new StreamReader(response1.GetResponseStream(), Encoding.ASCII);
//MessageBox.Show(sr.ReadToEnd());
HttpWebRequest request2 = (HttpWebRequest)WebRequest.Create(login_url);
request2.Method = "POST";
request2.CookieContainer = cookieJar;
request2.UserAgent = "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:25.0) Gecko/20100101 Firefox/25.0";
request2.KeepAlive = true;
request2.Timeout = 100000;
request2.AllowAutoRedirect = false;
request2.ContentType = "application/x-www-form-urlencoded";
byte[] byteArray2 = Encoding.ASCII.GetBytes(post_data);
request2.ContentLength = byteArray2.Length;
Stream dataStream2 = request2.GetRequestStream();
dataStream2.Write(byteArray2, 0, byteArray2.Length);
dataStream2.Close();
HttpWebResponse response2 = (HttpWebResponse)request2.GetResponse();
//MessageBox.Show(((HttpWebResponse)response2).StatusDescription);
dataStream2 = response2.GetResponseStream();
StreamReader reader2 = new StreamReader(dataStream2);
string report2 = reader2.ReadToEnd();
//MessageBox.Show(report2);
reader2.Close();
dataStream2.Close();
response2.Close();
HttpWebRequest request3 = (HttpWebRequest)WebRequest.Create(admin_url);
request3.Method = "GET";
request3.CookieContainer = cookieJar;
request3.ContentType = "text/html";
request3.UserAgent = "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:25.0) Gecko/20100101 Firefox/25.0";
request3.KeepAlive = true;
request3.Timeout = 100000;
request3.AllowAutoRedirect = true;
HttpWebResponse response3 = (HttpWebResponse)request3.GetResponse();
StreamReader sr3 = new StreamReader(response3.GetResponseStream(), Encoding.ASCII);
string report3 = sr3.ReadToEnd();
return report3;
}
}
}
But unfortunately in responce I get only HTML source code of login page and it seems that cookies don't contain session ID. All requests which I perform after that code also return HTML source of login page so I can assume that it does not login correctly.
Hi I am using a facebook posts feeding application which will read all the new posts from my facebook page and will save to my sharepoint list. Earlier, i.e, before June it has worked properly it feeds all the posts from fabebook to my Share point list. But nowadays its throws an error while getting response from the Facebook authorize url. I don't know what went wrong. Please help me if you guys have any suggestion to resolve this issue. I am appending my code part below.
private void btnSendToList_Click(object sender, EventArgs e)
{
try
{
if (ConfigurationSettings.AppSettings["fbClientID"] != null)
strClientID = ConfigurationSettings.AppSettings["fbClientID"];
if (ConfigurationSettings.AppSettings["fbRedirectURL"] != null)
strURL = ConfigurationSettings.AppSettings["fbRedirectURL"];
if (ConfigurationSettings.AppSettings["fbCltSecret"] != null)
strCltSecret = ConfigurationSettings.AppSettings["fbCltSecret"];
if (ConfigurationSettings.AppSettings["fbUserId"] != null)
strUserId = ConfigurationSettings.AppSettings["fbUserId"];
if (ConfigurationSettings.AppSettings["SPSiteURL"] != null)
strSiteURL = ConfigurationSettings.AppSettings["SPSiteURL"];
CookieCollection cookies = new CookieCollection();
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("https://www.facebook.com");
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.121 Safari/535.2";
request.CookieContainer = new CookieContainer();
request.CookieContainer.Add(cookies);
//Get the response from the server and save the cookies from the first request..
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
cookies = response.Cookies;
string getUrl = "https://www.facebook.com/login.php?login_attempt=1";
string postData = String.Format("email={0}&pass={1}", "testuser#gmail.com", "test123$"); // Masking credentials.
getRequest = (HttpWebRequest)WebRequest.Create(getUrl);
getRequest.CookieContainer = request.CookieContainer;
getRequest.CookieContainer.Add(cookies);
getRequest.Method = WebRequestMethods.Http.Post;
getRequest.UserAgent = "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.121 Safari/535.2";
getRequest.AllowWriteStreamBuffering = true;
getRequest.ProtocolVersion = HttpVersion.Version11;
getRequest.AllowAutoRedirect = true;
getRequest.ContentType = "application/x-www-form-urlencoded";
byteArray = Encoding.ASCII.GetBytes(postData);
getRequest.ContentLength = byteArray.Length;
newStream = getRequest.GetRequestStream(); //open connection
newStream.Write(byteArray, 0, byteArray.Length); // Send the data.
getResponse = (HttpWebResponse)getRequest.GetResponse();
getRequest = (HttpWebRequest)WebRequest.Create(string.Format("https://graph.facebook.com/oauth/authorize?client_id={0}&redirect_uri={1}", strClientID, "http://www.facebook.com/connect/login_success.html"));
getRequest.Method = WebRequestMethods.Http.Get;
getRequest.CookieContainer = request.CookieContainer;
getRequest.CookieContainer.Add(cookies);
getRequest.UserAgent = "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.121 Safari/535.2";
getRequest.AllowWriteStreamBuffering = true;
//getRequest.ProtocolVersion = HttpVersion.Version11;
getRequest.AllowAutoRedirect = true;
getRequest.ContentType = "application/x-www-form-urlencoded";
getResponse = (HttpWebResponse)getRequest.GetResponse();
The above line throws WebExceptopn which is teeling like Process has timed out.
strAccessToken = getResponse.ResponseUri.Query;
strAccessToken = strAccessToken.Replace("#_=_", "");
strAccessToken = strAccessToken.Replace("?code=", "");
newStream.Close();
if (!string.IsNullOrEmpty(strAccessToken))
strCode = strAccessToken;
if (!string.IsNullOrEmpty(strCode) && !string.IsNullOrEmpty(strClientID) &&
!string.IsNullOrEmpty(strURL) && !string.IsNullOrEmpty(strCltSecret) &&
!string.IsNullOrEmpty(strUserId))
{
SaveToList();
}
}
catch (Exception ex)
{
LogError(ex);
}
}
Hi Finally i got the solution.
Here i am using getResponse = (HttpWebResponse)getRequest.GetResponse();
The problem is we can use only one HttpWebResponse at a time. In my case i am using the same object in two times without any disposing and closing. That's make me the error.
So I updated my code like this.
byteArray = Encoding.ASCII.GetBytes(postData);
getRequest.ContentLength = byteArray.Length;
newStream = getRequest.GetRequestStream(); //open connection
newStream.Write(byteArray, 0, byteArray.Length); // Send the data.
getResponse = (HttpWebResponse)getRequest.GetResponse();
getresponse.Close();
This solved my error. Thanks