We Keep Coding

C# HTTPWebRequest.GetResponse() strips out query string

I'm almost certain that I'm just misunderstanding how query strings should be passed in using C# HTTPWebRequest/Response, but I can't figure out why when I call (HTTPwebResponse)request.GetResponse(); it strips out the query portion and only gives the resulting HTML of the base URL. What am I doing wrong? Note that I generalized the URL for simplicity but kept the same query string just in case the structure of that is the problem.
string urlQueryString = "https://baseUrl.com/folder1/folder2?status_filter_mode=approved";
var targetUri = Uri.EscapeUriString(urlQueryString);
request = HttpWebRequest.Create(targetUri) as HttpWebRequest;
request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; Touch; rv:11.0) like Gecko";
request.Method = "GET";
//set the cookie in the request header
request.Headers.Add("Cookie", cookiedata);
response = (HttpWebResponse)request.GetResponse();
using (Stream stream = response.GetResponseStream())
{
using (StreamReader reader = new StreamReader(stream))
{
htmlResult = reader.ReadToEnd();
}
}
response.Close();

C# HttpWebRequest response times out eventough works in browser

I spent several hours debuging the code, playing with Fiddler and googling, but still no luck, so hopefully you will help me.
I am trying to get the source of http://www.finishline.com. The catch is, the HttpWebRequest works in some regions (like here in Slovakia), but doesn't work in USA what I need to achieve.
For USA the request.GetResponse() just time outs. I have tried countless headers combinations, but without success. Can you please help? Thank you
var request = (HttpWebRequest)WebRequest.Create("http://www.finishline.com");
request.CookieContainer = new CookieContainer();
request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36";
request.AutomaticDecompression = (DecompressionMethods.GZip | DecompressionMethods.Deflate);
request.Headers.Add("Accept-Encoding", "gzip, deflate");
request.Accept = " text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
request.Headers.Add("Upgrade-Insecure-Requests", "1");
request.Headers.Add("Accept-Language", "sk,cs;q=0.8,en-US;q=0.5,en;q=0.3");
request.KeepAlive = true;
request.Headers.Add("Cache-Control", "max-age=0");
var responseText = "";
using (var response = request.GetResponse())
{
var httpWebResponse = response.GetResponseStream();
using (var sr = new StreamReader(httpWebResponse))
{
responseText = sr.ReadToEnd();
}
}

There are two kind of timeouts. Client timeout and server timeout. Have you tried doing something like this:
request.Timeout = Timeout.Infinite;
request.KeepAlive = true;
So, your GetResponse never get timedout you can check with it.
Or
using (HttpWebResponse response = (HttpWebResponse) request.GetResponse())
{
var httpWebResponse = response.GetResponseStream();
using (var sr = new StreamReader(httpWebResponse))
{
responseText = sr.ReadToEnd();
}
}
Try this method to fix your issue.

HttpWebRequest.GetResponse Operation has timed out

I'm trying to get simple gzip encoded html response from a website and it keeps getting time out, following is my code:
HttpWebRequest httpClient = (HttpWebRequest)WebRequest.Create(url);
httpClient.Method = "GET";
httpClient.Accept = "text/html, application/xhtml+xml, */*";
httpClient.Headers.Add("Accept-Encoding: gzip, deflate");
httpClient.Headers.Add("Accept-Language: en-US");
httpClient.Headers.Add("DNT: 1");
httpClient.ProtocolVersion = HttpVersion.Version10;
httpClient.KeepAlive = true;
httpClient.Timeout = System.Threading.Timeout.Infinite;
httpClient.CookieContainer = cookieJar;
String responseAsText;
using (HttpWebResponse response = (HttpWebResponse)httpClient.GetResponse())
{
System.IO.StreamReader sr;
if (response.ContentEncoding.Equals("gzip"))
{
sr = new StreamReader(new GZipStream(response.GetResponseStream(), CompressionMode.Decompress));
}
else
{
sr = new System.IO.StreamReader(response.GetResponseStream());
}
responseAsText = sr.ReadToEnd();
}
The url I'm trying to hit is "https client.schwab.com/Login/SignOn/CustomerCenterLogin.aspx"
This works perfectly fine in the Browser, using Fiddler I viewed the browser's Request header and since its Transfer-Encoding: chunked, I have used HttpVersion10
I have also tried setting httpClient.Timeout = System.Threading.Timeout.Infinite, but it never gets back with a response, however in browser the response gets in few seconds.
Please someone help me in achieving this.

probably you can try setting Agent property, so it doesn't recognize you as a bot.

I think Nero has answered your question ..
Try adding these Lines in your code..
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101 Firefox/24.0";

.NET HTTP POST Method - Cookies AirFrames.org

Trying to use C# to POST HttpWebRequest into airframes.org for aircraft information. This is the code I use for many other POST request with no problems (used with other urls), but it / I am not able to load the airframes.org page using ICAO24 number (A64294).
var cookies = new CookieContainer();
ServicePointManager.Expect100Continue = false;
var request = (HttpWebRequest)WebRequest.Create("http://www.airframes.org/");
request.CookieContainer = cookies;
request.Method = "POST";
request.KeepAlive = true;
request.ContentType = "application/x-www-form-urlencoded";
using (var requestStream = request.GetRequestStream())
using (var writer = new StreamWriter(requestStream))
{
writer.Write("reg=&selcal=&icao24=A64294&submit=submit");
}
using (var responseStream = request.GetResponse().GetResponseStream())
using (var reader = new StreamReader(responseStream))
{
var result = reader.ReadToEnd();
Console.WriteLine(result);
}

Note that the site has a no-bots policy, which is why your request won't work.
That being said, if you still wish to request the page, adding a user-agent string (so your request looks like it came from a browser) does the trick:
request.UserAgent = "Mozilla/6.0 (Windows NT 6.2; WOW64; rv:16.0.1) Gecko/20121011 Firefox/16.0.1";
It is usually a good idea to respect the policies of a site. The code above is merely for educational purposes.

How do I duplicate this JavaScript in C#

Here is the JavaScript
function ChangeVol(id)
{
document.form.selectFS_devId.value = id;
document.form.selectFS_currentNameSpace.value = "";
document.form.submit();
}
function ChangeEvsVol(id, vNodeId)
{
document.form.selectFS_evsId.value = vNodeId;
document.form.selectFS_currentNameSpace.value = "";
ChangeVol(id);
}
document.form.selectFS_devId.value = "all"
document.form.selectFS_evsId.value = "2"
Here is the current C# code I'm using
Uri url = new Uri("https://mgr/app");
HttpWebRequest request = null;
ServicePointManager.ServerCertificateValidationCallback =
((sender, certificate, chain, sslPolicyErrors) => true);
CookieContainer cookieJar = new CookieContainer();
request = (HttpWebRequest)WebRequest.Create(url);
request.CookieContainer = cookieJar;
request.Method = "GET";
HttpStatusCode responseStatus;
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
responseStatus = response.StatusCode;
url = request.Address;
}
if (responseStatus == HttpStatusCode.OK)
{
UriBuilder urlBuilder = new UriBuilder(url);
urlBuilder.Path =
urlBuilder.Path.Remove(urlBuilder.Path.LastIndexOf('/')) +
"/j_security_check";
request = (HttpWebRequest)WebRequest.Create(urlBuilder.ToString());
request.Referer = url.ToString();
request.CookieContainer = cookieJar;
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded";
using (Stream requestStream = request.GetRequestStream())
using (StreamWriter requestWriter = new StreamWriter(requestStream, Encoding.ASCII))
{
string postData = "j_username=user&j_password=user&submit=Send";
requestWriter.Write(postData);
}
string responseContent = null;
string myTargetString = "https://mgr/app/action/storage.VivolAction/eventsubmit_dopreparevivollist/ignored/f5/true";
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
using (Stream responseStream = response.GetResponseStream())
using (StreamReader responseReader = new StreamReader(responseStream))
{
responseContent = responseReader.ReadToEnd();
}
Console.WriteLine(responseContent);
request = (HttpWebRequest)WebRequest.Create(myTargetString);
request.Method = "GET";
request.CookieContainer = cookieJar;
using (HttpWebResponse responsedownload = (HttpWebResponse)request.GetResponse())
using (Stream responseStream = responsedownload.GetResponseStream())
using (StreamReader responseReader = new StreamReader(responseStream))
{
responseContent = responseReader.ReadToEnd();
}
Console.WriteLine(responseContent);
the problem is the string myTargetString doesn't load the javascript params, if i could duplicate those params in the URL would be awesome, if not, what would I need to do to submit those in a post request like I do above in the StreamWriter?
using (StreamWriter requestWriter = new StreamWriter(requestStream, Encoding.ASCII))
{
string postData = "j_username=user&j_password=user&submit=Send";
requestWriter.Write(postData);
}
What I mean by in the url is perhaps something like:
https://mgr/app/action/storage.VivolAction/eventsubmit_dopreparevivollist/ignored?&evsId=1&devId=all&currentpagenumberbottom=1&filtername=&currentpagenumber=1&quotaactionlink=/mgr/app/action/storage.VivolQuotaAction&ascending=true&currentpagesize=20&ignoreErrorMessages=true&pageindex=1&sortby=name&filterpath=
Fiddler provided me with this
POST https://mgr/app/action/storage.SelectFileSystemAction/eventsubmit_doprocessselectfilesystem/ignored
Accept: image/gif, image/jpeg, image/pjpeg, image/pjpeg, application/x-shockwave-flash, application/xaml+xml, application/x-ms-xbap, application/x-ms-application, application/vnd.ms-xpsdocument, application/vnd.ms-excel, application/vnd.ms-powerpoint, application/msword, */*
Referer: https://mgr/app/action/storage.SelectFileSystemAction /eventsubmit_doprepareselectfilesystem/ignored
Accept-Language: en-us
User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0)
Content-Type: application/x-www-form-urlencoded
Accept-Encoding: gzip, deflate
Host: arc
Content-Length: 378
Connection: Keep-Alive
Cache-Control: no-cache
Cookie: jid=asdsad ; jsso = asdas2sa
op=&selectFS_devId=all&selectFS_previous_template=&selectFS_evsId=2&selectFS_currentNameSpace=&selectFS_action_class=storage.VivolAction&selectFS_action_method=doPreparevivollist&selectFS_uniqueId=13655b454e3951462f&selectFS_dont_alter_current=false&selectFS_disableReplicationTargets=true&selectFS_disableReadCache=true&selectFS_disableWorm=false&selectFS_disableUnmounted=true
I can see the electFS_devId=all and selectFS_evsId=2 in there, I need to change the EVSID but I'm not sure how to contruct the URL. Yes I changed the cookie id's

Your javascript is just setting the form values for what are probably hidden fields on the form before performing the submit. You'll need to do a POST request, the same way that you do for the login.
Look at the action value on the form tag in your HTML to determine where to submit the data and put the following form items into your postData:
selectFS_devId
selectFS_currentNameSpace
selectFS_evsId
You can use something like:
string postData = string.Format("selectFS_currentNameSpace={0}&selectFS_evsId={1}&selectFS_devId={2}", "", "2", "all");
Looks like this will get involved, because it appears to be doing some form of session or transaction tracking using selectFS_uniqueId you'll likely have to do a GET operation first and then extract that value from the form. Also, notice that your submit location, just like with the prior j_security_check, isn't going to the same location for the POST (doprocess) as the GET (doprepare) that retrieves the form.
GET
https://mgr/app/action/storage.SelectFileSystemAction/eventsubmit_doprepareselectfilesystem/ignored
POST
https://mgr/app/action/storage.SelectFileSystemAction/eventsubmit_doprocessselectfilesystem/ignored
Take another look at your post values here too. Clearly there is a command being issued with a class (selectFS_action_class) and method (selectFS_action_method) and likely nothing will be done if they aren't specified.
op=
&selectFS_devId=all
&selectFS_previous_template=
&selectFS_evsId=2
&selectFS_currentNameSpace=
&selectFS_action_class=storage.VivolAction
&selectFS_action_method=doPreparevivollist
&selectFS_uniqueId=13655b454e3951462f
&selectFS_dont_alter_current=false
&selectFS_disableReplicationTargets=true
&selectFS_disableReadCache=true
&selectFS_disableWorm=false
&selectFS_disableUnmounted=true
Rather than writing all this stuff to emulate a user doing operations through the web interface, have you checked with F5 to see if they have web services that you can call to do this?

Maybe the ploblem is that, the parameters in this form "j_username=user&j_password=user&submit=Send" are parameters for HTTP "GET" not "POST" you can try get the parameters on this way.
C#
var operacion = context.Request.Form[0] "POST"
var operacion = context.Request.Params[0];
or
Request.Querystring("parameterName") for "GET"
or in javascript you can use $_GET or $_POST

I think your question is way to long for what you are asking. plus you didn't show any code about the posting in the browser.
it seems like you just want to know how to get a form post to work...I noticed you aren't setting the contentlength property...that might have something to do with it. also, check out this "post"...
How to post data to specific URL using WebClient in C#