We Keep Coding

Why is my C# HttpWebRequest not timing out after 4 seconds

My application is downloading many web pages and files and I don't want it to ever spend more than 4 seconds trying to download a webpage or file. I can't get the correct timeout settings to allow this.
What setting an I missing here?
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(websiteURL);
request.Timeout = 4000;
request.KeepAlive = false;
request.AllowAutoRedirect = true;
request.ProtocolVersion = HttpVersion.Version11;
request.Accept = "text/html, application/xhtml+xml, application/xml; q=0.9, image/webp, */*; q=0.8";
request.Headers["Accept-Language"] = "en-GB,en-US;q=0.7,en;q=0.3";
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.97 Safari/537.36";
request.ReadWriteTimeout = 4000;
request.AllowWriteStreamBuffering = false;
response = (HttpWebResponse)request.GetResponse();
Stream dataStream = response.GetResponseStream();
StreamReader reader = new StreamReader(dataStream);
reader.BaseStream.ReadTimeout = 4000;
dataStream.WriteTimeout = 4000;
dataStream.ReadTimeout = 4000;
responseFromServer = reader.ReadToEnd();

You should use httpClient instead of HttpWebRequest.
I used httpClient and timeout worked fine.
Try this:
using (var httpClient = new HttpClient())
{
httpClient.Timeout = new TimeSpan(0, 0, 4);
Stream response = await httpClient.GetStreamAsync("https://www.website.com/");
StreamReader reader = new StreamReader(response);
}
HttpClient vs HttpWebRequest

Need to create httpwebrequest for sending api post in c#

I have tried to create a httpwebrequest for post request to api site. The header details below;enter image description here
and I have tried below code;
HttpWebRequest webRequest = (HttpWebRequest)WebRequest.Create("https://www.etc.com/c/search");
//webRequest.Credentials = cache;
//webRequest.PreAuthenticate = true;
//header description
webRequest.Method = "POST";
webRequest.Connection = "keep-alive";
//webRequest.KeepAlive = true;
byte[] byteArray = Encoding.UTF8.GetBytes("{\"contains\":true,\"empName\":RAGS TO RACHES,\"injuryDate\":04/13/2018,\"state\":UT,\"stCd\":43}");
webRequest.Accept = "q=0.8;application/json;q=0.9";
webRequest.UserAgent = #"Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:59.0) Gecko/20100101 Firefox/59.0";
webRequest.ContentType = "application/json";
webRequest.ContentLength = byteArray.Length;
webRequest.Referer = "https://www.etc.com/c/search";
webRequest.Headers.Add("Accept-Encoding: gzip, deflate, br");
webRequest.Headers.Add("Accept-Language: en-US,en;q=0.5");
//webRequest.Headers.Add("Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3");
using (Stream webpageStream = webRequest.GetRequestStream())
{
webpageStream.Write(byteArray, 0, byteArray.Length);
}
Not getting resposne.

C# HttpWebRequest response times out eventough works in browser

I spent several hours debuging the code, playing with Fiddler and googling, but still no luck, so hopefully you will help me.
I am trying to get the source of http://www.finishline.com. The catch is, the HttpWebRequest works in some regions (like here in Slovakia), but doesn't work in USA what I need to achieve.
For USA the request.GetResponse() just time outs. I have tried countless headers combinations, but without success. Can you please help? Thank you
var request = (HttpWebRequest)WebRequest.Create("http://www.finishline.com");
request.CookieContainer = new CookieContainer();
request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36";
request.AutomaticDecompression = (DecompressionMethods.GZip | DecompressionMethods.Deflate);
request.Headers.Add("Accept-Encoding", "gzip, deflate");
request.Accept = " text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
request.Headers.Add("Upgrade-Insecure-Requests", "1");
request.Headers.Add("Accept-Language", "sk,cs;q=0.8,en-US;q=0.5,en;q=0.3");
request.KeepAlive = true;
request.Headers.Add("Cache-Control", "max-age=0");
var responseText = "";
using (var response = request.GetResponse())
{
var httpWebResponse = response.GetResponseStream();
using (var sr = new StreamReader(httpWebResponse))
{
responseText = sr.ReadToEnd();
}
}

There are two kind of timeouts. Client timeout and server timeout. Have you tried doing something like this:
request.Timeout = Timeout.Infinite;
request.KeepAlive = true;
So, your GetResponse never get timedout you can check with it.
Or
using (HttpWebResponse response = (HttpWebResponse) request.GetResponse())
{
var httpWebResponse = response.GetResponseStream();
using (var sr = new StreamReader(httpWebResponse))
{
responseText = sr.ReadToEnd();
}
}
Try this method to fix your issue.

Garbage text in GetResponseStream()

I keep getting garbage text like this
�\b\0\0\0\0\0\0�\a`I�%&/m�{J�J��t�\b�`$ؐ#������iG#
in my html variable at the end of this function. As you can see, by the commented out code, I have tried both methods of inserting Cookies.
I am very new to HTTPwebRequest / Response methods like this. But from everything I can find on the web I am setting up my method correctly. I would love some help if possible.
Also, when using Fiddler to decode my requests, not all of my cookies are getting sent. I am only sending 1 utma, 1 utmb, 1 utmc, and 1 utmz when my code runs. However, when I log into the site normally, i receive 2 utma, 1 utmb, 2 utmc, and 2 utmz.
I feel that this is the source of my connecting issues but I am not sure.
static void FillCookieJar()
{
Console.WriteLine("Filling cookie jar...\r\n");
try
{
string parameters = "SUPER LONG POST DATA found from TEXTVIEW in Fidler";
Uri target = new Uri("https://foo.bar.com/UserSignIn");
byte[] buffer = Encoding.ASCII.GetBytes(parameters);
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(target);
//Cookie chipOne = new Cookie("__utma", "XXXXXXXXX.XXXXXXXXX.XXXXXXXXX.XXXXXXXXX.XXXXXXXXX.XX") { Domain = target.Host };
//Cookie chipTwo = new Cookie("__utma", "XXXXXXXXX.XXXXXXXXX.XXXXXXXXX.XXXXXXXXX.XXXXXXXXX.X") { Domain = target.Host };
//Cookie chipThree = new Cookie("__utmb", "XXXXXXXXX.X.XX.XXXXXXXXX") { Domain = target.Host };
//Cookie chipFour = new Cookie("__utmc", "XXXXXXXXX") { Domain = target.Host };
//Cookie chipFive = new Cookie("__utmc", "XXXXXXXXX") { Domain = target.Host };
//Cookie chipSix = new Cookie("__utmz", "XXXXXXXXX.XXXXXXXXX.X.X.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=(not%20provided)") { Domain = target.Host };
//Cookie chipSeven = new Cookie("__utmz", "XXXXXXXXX.XXXXXXXXX.XX.X.utmcsr=titlesource.com|utmccn=(referral)|utmcmd=referral|utmcct=/") { Domain = target.Host };
//cookieJar.Add(chipOne);
//cookieJar.Add(chipTwo);
//cookieJar.Add(chipThree);
//cookieJar.Add(chipFour);
//cookieJar.Add(chipFive);
//cookieJar.Add(chipSix);
//cookieJar.Add(chipSeven);
request.Headers.Add("__utma", "XXXXXXXXX.XXXXXXXXX.XXXXXXXXX.XXXXXXXXX.XXXXXXXXX.XX");
request.Headers.Add("__utma", "XXXXXXXXX.XXXXXXXXX.XXXXXXXXX.XXXXXXXXX.XXXXXXXXX.X");
request.Headers.Add("__utmb", "XXXXXXXXX.X.XX.XXXXXXXXX");
request.Headers.Add("__utmc", "XXXXXXXXX");
request.Headers.Add("__utmc", "XXXXXXXXX");
request.Headers.Add("__utmz", "XXXXXXXXX.XXXXXXXXX.X.X.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=(not%20provided)");
request.Headers.Add("__utmz", "XXXXXXXXX.XXXXXXXXX.XX.X.utmcsr=titlesource.com|utmccn=(referral)|utmcmd=referral|utmcct=/");
request.CookieContainer = cookieJar;
request.Method = WebRequestMethods.Http.Post;
request.KeepAlive = true;
request.Accept = "*/*";
request.Headers.Add("Cache-Control", "no-cache");
request.Headers.Add("Accept-Encoding: gzip,deflate,sdch");
request.Headers.Add("Accept-Language: en-US,en;q=0.8");
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.114 Safari/537.36";
request.Headers.Add("X-Requested-With: XMLHttpRequest");
request.ContentType = "application/x-www-form-urlencoded; charset=UTF-8";
request.Referer = "https://foo.bar.com/UserSignIn";
request.Headers.Add("Origin", "https://foo.bar.com");
request.Headers.Add("X-MicrosoftAjax", "Delta=true");
request.ContentLength = buffer.Length;
Stream PostData = request.GetRequestStream();
PostData.Write(buffer, 0, buffer.Length);
PostData.Close();
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Stream Answer = response.GetResponseStream();
StreamReader _Answer = new StreamReader(Answer);
String html = _Answer.ReadToEnd();
response.Close();
}
catch (Exception ex)
{
Console.WriteLine("Error FillingCookieJar");
}
}

You need to decompress the gzipped stream to get back the plain text, which is likely in UTF-8 rather than ASCII (look for a charset attribute on the response's Content-Type header).
You can use the AutomaticDecompression property to automatically decompress the content.
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(target);
request.AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip;
The url you see is UrlEncoded. You can use HttpUtility.UrlDecode to get the URL to look like https://foo.bar.com/Vendor.
string decoded = HttpUtility.UrlDecode(html);

getting response from Facebook is getting timed out

Hi I am using a facebook posts feeding application which will read all the new posts from my facebook page and will save to my sharepoint list. Earlier, i.e, before June it has worked properly it feeds all the posts from fabebook to my Share point list. But nowadays its throws an error while getting response from the Facebook authorize url. I don't know what went wrong. Please help me if you guys have any suggestion to resolve this issue. I am appending my code part below.
private void btnSendToList_Click(object sender, EventArgs e)
{
try
{
if (ConfigurationSettings.AppSettings["fbClientID"] != null)
strClientID = ConfigurationSettings.AppSettings["fbClientID"];
if (ConfigurationSettings.AppSettings["fbRedirectURL"] != null)
strURL = ConfigurationSettings.AppSettings["fbRedirectURL"];
if (ConfigurationSettings.AppSettings["fbCltSecret"] != null)
strCltSecret = ConfigurationSettings.AppSettings["fbCltSecret"];
if (ConfigurationSettings.AppSettings["fbUserId"] != null)
strUserId = ConfigurationSettings.AppSettings["fbUserId"];
if (ConfigurationSettings.AppSettings["SPSiteURL"] != null)
strSiteURL = ConfigurationSettings.AppSettings["SPSiteURL"];
CookieCollection cookies = new CookieCollection();
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("https://www.facebook.com");
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.121 Safari/535.2";
request.CookieContainer = new CookieContainer();
request.CookieContainer.Add(cookies);
//Get the response from the server and save the cookies from the first request..
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
cookies = response.Cookies;
string getUrl = "https://www.facebook.com/login.php?login_attempt=1";
string postData = String.Format("email={0}&pass={1}", "testuser#gmail.com", "test123$"); // Masking credentials.
getRequest = (HttpWebRequest)WebRequest.Create(getUrl);
getRequest.CookieContainer = request.CookieContainer;
getRequest.CookieContainer.Add(cookies);
getRequest.Method = WebRequestMethods.Http.Post;
getRequest.UserAgent = "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.121 Safari/535.2";
getRequest.AllowWriteStreamBuffering = true;
getRequest.ProtocolVersion = HttpVersion.Version11;
getRequest.AllowAutoRedirect = true;
getRequest.ContentType = "application/x-www-form-urlencoded";
byteArray = Encoding.ASCII.GetBytes(postData);
getRequest.ContentLength = byteArray.Length;
newStream = getRequest.GetRequestStream(); //open connection
newStream.Write(byteArray, 0, byteArray.Length); // Send the data.
getResponse = (HttpWebResponse)getRequest.GetResponse();
getRequest = (HttpWebRequest)WebRequest.Create(string.Format("https://graph.facebook.com/oauth/authorize?client_id={0}&redirect_uri={1}", strClientID, "http://www.facebook.com/connect/login_success.html"));
getRequest.Method = WebRequestMethods.Http.Get;
getRequest.CookieContainer = request.CookieContainer;
getRequest.CookieContainer.Add(cookies);
getRequest.UserAgent = "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.121 Safari/535.2";
getRequest.AllowWriteStreamBuffering = true;
//getRequest.ProtocolVersion = HttpVersion.Version11;
getRequest.AllowAutoRedirect = true;
getRequest.ContentType = "application/x-www-form-urlencoded";
getResponse = (HttpWebResponse)getRequest.GetResponse();
The above line throws WebExceptopn which is teeling like Process has timed out.
strAccessToken = getResponse.ResponseUri.Query;
strAccessToken = strAccessToken.Replace("#_=_", "");
strAccessToken = strAccessToken.Replace("?code=", "");
newStream.Close();
if (!string.IsNullOrEmpty(strAccessToken))
strCode = strAccessToken;
if (!string.IsNullOrEmpty(strCode) && !string.IsNullOrEmpty(strClientID) &&
!string.IsNullOrEmpty(strURL) && !string.IsNullOrEmpty(strCltSecret) &&
!string.IsNullOrEmpty(strUserId))
{
SaveToList();
}
}
catch (Exception ex)
{
LogError(ex);
}
}

Hi Finally i got the solution.
Here i am using getResponse = (HttpWebResponse)getRequest.GetResponse();
The problem is we can use only one HttpWebResponse at a time. In my case i am using the same object in two times without any disposing and closing. That's make me the error.
So I updated my code like this.
byteArray = Encoding.ASCII.GetBytes(postData);
getRequest.ContentLength = byteArray.Length;
newStream = getRequest.GetRequestStream(); //open connection
newStream.Write(byteArray, 0, byteArray.Length); // Send the data.
getResponse = (HttpWebResponse)getRequest.GetResponse();
getresponse.Close();
This solved my error. Thanks