When I try to fetch the content of this URL
http://www.yellowpages.com.au/qld/gatton/a-12000029-listing.html
using System.Net;
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(link);
request.AllowAutoRedirect = true;
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Stream resStream = response.GetResponseStream();
StreamReader objSR;
objSR = new StreamReader(resStream, System.Text.Encoding.GetEncoding("utf-8"));
string sResponse = objSR.ReadToEnd();
I don't get any response from the server. Please help me find out why this happens.
Thanks in advance!
It may well be looking at the user agent and refusing to serve content to a client that doesn't identify itself. Try setting the UserAgent property on your request object.
Looks to me like that site is checking the referrer url and may be serving up empty content if an invalid referrer is specified.
Try setting request.Referer = "http://www.google.com";. Experiment with the referrer to see if that changes the response. I'd also try the UserAgent property as Matthew suggested.
I had the same problem and the cause was that I previously had set the method to HEAD and in later revisions had the need to parse the body.
Related
Using fiddler to observe this URL:
http://opencaselist.paperlessdebate.com/bin/AllDocs?view=attachments#format=json?|t=allattachments&p=1&l=10&s=filename&d=asc
I find a nice JSON response like this
How do I get this response into a string that I can save in a txt file using C#? Is there a way to turn an HTTP Web Response to a string? Is there something is NewtonSoft JSON that can help me? Are there particular terms that will help me google this more effectively?
Every time I try I just get an HTML version of the web-page at the link and not the JSON data I'm trying to get:
string url = "http://opencaselist.paperlessdebate.com/bin/AllDocs?view=attachments#|t=allattachments&p=1&l=10&s=filename&d=asc";
HttpWebRequest httpWebRequest = (HttpWebRequest)WebRequest.Create(url);
httpWebRequest.Method = WebRequestMethods.Http.Get;
httpWebRequest.Accept = "text/json";
httpWebRequest = (HttpWebRequest)WebRequest.Create(url);
HttpWebResponse response = (HttpWebResponse)httpWebRequest.GetResponse();
response.Close();
Note that on that screenshot, there are some differences:
the URL is way different: AllAttachements vs AllDocs, but that's minor, I'm pointing it out "just in case"
the PARAMS are way different: the screenshot specifies xpage=plain&outputSyntax=plain and your code - not
the HEADERS are different: your code has Accept=text\json while screenshot has Accept: text/javascript
Have you tried using the same params and headers?
EDIT: also, I've opened up the page from your code, and it actually is a page. After loading, it generates additional requests to
http://opencaselist.paperlessdebate.com/bin/get/XWiki/AllAttachmentsResults?xpage=plain&outputSyntax=plain&offset=1&limit=10&reqNo=1&sort=filename&dir=asc
which, if you download, results in JSON data. No headers at all, simple GET. I've just got the JSON data by simply pasting that URL into Chrome.. I think that you simply use wrong URL.
If I copy and past the following URL in the browser, I get a response URL with a query string sessionid in it:
https://abc.abcdefg.com/abcd/sessionServlet
I am trying to capture that response URL and session id in my code behind in .net:
string url = "https://abc.abcdefg.com/abcd/sessionServlet";
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Label1.Text = response.ResponseUri.ToString();
the response.ResponseUri contains my original URL, but not the response URL I get back from the sessionServlet.
Could anyone help me? Thank you in advance.
Looking at your comment about the http://devserver/myproject/login.aspx?sessionid=1341351j1oij4o1i3o13i5ho1i3j4134o URL appearing the browser from vising the URL https://abc.abcdefg.com/abcd/sessionServlet, you're probably being redirected using HTTP 301 or 302.
If so, I'd add request.AllowAutoRedirect = true MSDN which will allow your web request to follow that redirect. Then response.ResponseUri.Query should have the querystring that you're looking for.
I want to visit a web page (it has to be accessed, nothing needs to be read, modified, etc. Just accessed). I don't want to use webbrowser.
Just do a cURL GET request.
curl http://example.com/
And if you want to use C#, then
using System.Net;
string url = "https://www.example.com/";
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Stream resStream = response.GetResponseStream();
Also, you can use Fiddler to send request to the remote server (it is very helpful for service debuging).
try to use WebClient class:
for example:
WebClient client = new WebClient ();
string reply = client.DownloadString (address);
Before everyone gets upset that this has been answered. I have scoured the web looking for how to do this and have tried a number of methods. Login to website, via C# and How to programmatically log in to a website to screenscape? Both of these were helpful but I cannot figure out why I cannot get past the login page. Here is my code:
string url = "https://www.advocare.com/login.aspx";
string url2 = "https://url.after.login";
HttpWebRequest wReq = WebRequest.Create(url) as HttpWebRequest;
wReq.KeepAlive = true;
wReq.Method = "POST";
wReq.AllowAutoRedirect = false;
wReq.ContentType = "application/x-www-form-urlencoded";
string postData = "ctl00$cphContent$txtUserName=Username&ctl00$cphContent$txtPassword=Password";
byte[] dataBytes = UTF8Encoding.UTF8.GetBytes(postData);
wReq.ContentLength = dataBytes.Length;
using (Stream postStream = wReq.GetRequestStream())
{
postStream.Write(dataBytes, 0, dataBytes.Length);
}
HttpWebResponse wResp = wReq.GetResponse() as HttpWebResponse;
string pageSource;
wReq = WebRequest.Create(url2) as HttpWebRequest;
wReq.CookieContainer = new CookieContainer();
wReq.CookieContainer.Add(wResp.Cookies);
HttpWebResponse wResp2 = wReq.GetResponse() as HttpWebResponse;
using (StreamReader sr = new StreamReader(wResp2.GetResponseStream()))
{
pageSource = sr.ReadToEnd();
}
Everytime I look at pageSource it is the HTML for the login.aspx page. I must be missing something here. Maybe it's not taking the cookie, I don't know. One question I have aside from, why doesn't this work, is in the string postData = "". Are those suppose to be the name or id portion of the html tag? Any help on this is greatly appreciated as I am stumped and will have to find a different way. I would like to continue with the WebRequest and WebResponse instead of using WebBrowser. If I can't, oh well. Thanks again for any help!
What are you trying to do besides login? If its like QAing a site programically, i would suggest using selenium andcreate a c# app based off of that. If u want i can post a link to a base project for a selenium based project.
Don't necessarily view the page source, but look at the actual HTTP POST. Install a HTTP proxy such as Fiddler and then re-visit the page you are trying to emulate. Complete the HTTP POST request, and check out the results produced in the proxy. From there you'll be able to see the actual parameters, cookies, headers, etc. that are being passed and you can then attempt to replicate this in your code. It's often easy to miss something when simply viewing the HTML source but monitoring the network traffic is pretty straight forward.
I have a know a website that contains an open database of the result of an academic test.
http://nts.org.pk/NTSWeb/PPL_30Sep2012_Result/search.asp
I am expert with C# but newbie to web development.
Usually, using web browser, we can enter and roll number and server sends back the result. E.G. Use my Roll Num: 3912125
What I need to do is, use a C# application to communicate this roll null number and get anything, of my result. (any string is excepted, I will parse out my result from that string.)
How do I send query? when I don't know a list of possible query strings.
I tried this code:
string queryString = "RollNo=3912125";
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(#"http://nts.org.pk/NTSWeb/PPL_30Sep2012_Result/search.asp");
request.UseDefaultCredentials = true;
request.ContentType = "application/x-www-form-urlencoded";
request.Method = "POST";
byte[] requestBytes = Encoding.UTF8.GetBytes(queryString);
request.ContentLength = requestBytes.Length;
using (var requestStream = request.GetRequestStream())
{
requestStream.Write(requestBytes, 0, requestBytes.Length);
requestStream.Close();
}
WebResponse response = request.GetResponse();
textBox1.AppendText(((HttpWebResponse)response).StatusDescription);
Stream dataStream = response.GetResponseStream();
StreamReader reader = new StreamReader(dataStream);
string responseFromServer = reader.ReadToEnd();
textBox1.AppendText(responseFromServer);
reader.Close();
dataStream.Close();
response.Close();
You have to append the querystring to the url like this:
string queryString = "RollNo=3912125";
string url = String.Format(#"http://foo/search.asp?{0}", queryString);
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
You should take a look at the code in my answer to C# https login and download file. It gives a good demonstration of how to perform a POST request. As far as knowing what's valid to use for the query-formatted string in your POST, it's simply a matter of looking for appropriate input elements in the page content. It looks like what you have (RollNo) is correct. You may, however, need to also add the submit button value to your request as well depending on how the server behaves, giving you something like. RollNo=3912125&submit=submit.
You're most of the way there. Your queryString should look like RollNo=3912125&Submit=+Search+. When you are calling WebRequest.Create, the Url should in fact be http://nts.org.pk/NTSWeb/PPL_30Sep2012_Result/result.asp.
The rest of your code should work, though the answer #JamieSee recommended to you has some very good advice about wrapping things in using blocks correctly