I have a know a website that contains an open database of the result of an academic test.
http://nts.org.pk/NTSWeb/PPL_30Sep2012_Result/search.asp
I am expert with C# but newbie to web development.
Usually, using web browser, we can enter and roll number and server sends back the result. E.G. Use my Roll Num: 3912125
What I need to do is, use a C# application to communicate this roll null number and get anything, of my result. (any string is excepted, I will parse out my result from that string.)
How do I send query? when I don't know a list of possible query strings.
I tried this code:
string queryString = "RollNo=3912125";
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(#"http://nts.org.pk/NTSWeb/PPL_30Sep2012_Result/search.asp");
request.UseDefaultCredentials = true;
request.ContentType = "application/x-www-form-urlencoded";
request.Method = "POST";
byte[] requestBytes = Encoding.UTF8.GetBytes(queryString);
request.ContentLength = requestBytes.Length;
using (var requestStream = request.GetRequestStream())
{
requestStream.Write(requestBytes, 0, requestBytes.Length);
requestStream.Close();
}
WebResponse response = request.GetResponse();
textBox1.AppendText(((HttpWebResponse)response).StatusDescription);
Stream dataStream = response.GetResponseStream();
StreamReader reader = new StreamReader(dataStream);
string responseFromServer = reader.ReadToEnd();
textBox1.AppendText(responseFromServer);
reader.Close();
dataStream.Close();
response.Close();
You have to append the querystring to the url like this:
string queryString = "RollNo=3912125";
string url = String.Format(#"http://foo/search.asp?{0}", queryString);
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
You should take a look at the code in my answer to C# https login and download file. It gives a good demonstration of how to perform a POST request. As far as knowing what's valid to use for the query-formatted string in your POST, it's simply a matter of looking for appropriate input elements in the page content. It looks like what you have (RollNo) is correct. You may, however, need to also add the submit button value to your request as well depending on how the server behaves, giving you something like. RollNo=3912125&submit=submit.
You're most of the way there. Your queryString should look like RollNo=3912125&Submit=+Search+. When you are calling WebRequest.Create, the Url should in fact be http://nts.org.pk/NTSWeb/PPL_30Sep2012_Result/result.asp.
The rest of your code should work, though the answer #JamieSee recommended to you has some very good advice about wrapping things in using blocks correctly
Related
I am making a rest call in which I get a HttpWebResponse that contains data. It seems the data is serialized, and I am trying to get the plain text of the request. I have been using the chrome extension Advanced Rest client, which when calling the same request it is able to display the text version of the json response.
From what I have read on here, you are required to deserialize into the expected object. However, it is pretty clear that chrome plugin has no idea about the object type and can still print out plain text.
Is it possible to do the same in c#?
HttpWebRequest request = WebRequest.Create(address) as HttpWebRequest;
request.Method = "POST";
request.ContentType = "application/json";
// [code removed for setting json data via stream writer
using (HttpWebResponse response = request.GetResponse() as HttpWebResponse)
{
// This is where I am trying to figure out how to get plain readable text out of response.GetResponseStream()
}
Edit: If I simply use a StreamReader to get the text from the response stream, I get a bunch of binary data and not the plain json text.
Edit: realized the problem had to do with compression. This can be closed.
I'm not sure if got it right, but you can get the response as a string doing this:
using (var sr = new StreamReader(response.GetResponseStream()))
{
text = sr.ReadToEnd();
}
Turned out my problem was due to compression. I realized the header contained "Content-Encoding: gzip" so I searched on how to unzip with gzip compression and then the text was proper json. Thanks all
I want to make an application that can interact with an existing website. The website has some search related text box fields that a user usually types stuff into and then clicks a button to go the next page. I'd like to automate this process by making a visual studio c# application. I'm not sure where to start, it seems like most tutorials out there are geared towards creating your own website that does stuff, not interacting with an existing one.
As far as I can tell, typing into the search and clicking the button doesn't seem to create a unique URL string (not like Google maps does), because the URL doesn't change when it take me to the search results. I'm not sure what type of TCP commands I need to send (I know about "GET", but that's about it).
Any pointers?
Entering text into the URL would be a GET request. TCP is just the underlying protocol beneath HTTP.
What you're looking for is a HTTP POST request.
Start with WebRequest.
// Create a request using a URL that can receive a post.
WebRequest request = WebRequest.Create ("http://www.contoso.com/PostAccepter.aspx ");
// Set the Method property of the request to POST.
request.Method = "POST";
// Create POST data and convert it to a byte array.
string postData = "This is a test that posts this string to a Web server.";
byte[] byteArray = Encoding.UTF8.GetBytes (postData);
// Set the ContentType property of the WebRequest.
request.ContentType = "application/x-www-form-urlencoded";
// Set the ContentLength property of the WebRequest.
request.ContentLength = byteArray.Length;
// Get the request stream.
Stream dataStream = request.GetRequestStream ();
// Write the data to the request stream.
dataStream.Write (byteArray, 0, byteArray.Length);
// Close the Stream object.
dataStream.Close ();
// Get the response.
WebResponse response = request.GetResponse ();
// Display the status.
Console.WriteLine (((HttpWebResponse)response).StatusDescription);
// Get the stream containing content returned by the server.
dataStream = response.GetResponseStream ();
// Open the stream using a StreamReader for easy access.
StreamReader reader = new StreamReader (dataStream);
// Read the content.
string responseFromServer = reader.ReadToEnd ();
// Display the content.
Console.WriteLine (responseFromServer);
// Clean up the streams.
reader.Close ();
dataStream.Close ();
response.Close ();
the postData variable contains the elements that you want to "insert" into the text boxes. It should give you a good start on this. I've been using this technique for all projects which involve website interaction.
Selenium which was suggested in the comments could be an option, only if you don't mind that a specific browser needs to be installed, as it utilizes it for this purpose. It's not a good solution if you want to deploy your application.
Before everyone gets upset that this has been answered. I have scoured the web looking for how to do this and have tried a number of methods. Login to website, via C# and How to programmatically log in to a website to screenscape? Both of these were helpful but I cannot figure out why I cannot get past the login page. Here is my code:
string url = "https://www.advocare.com/login.aspx";
string url2 = "https://url.after.login";
HttpWebRequest wReq = WebRequest.Create(url) as HttpWebRequest;
wReq.KeepAlive = true;
wReq.Method = "POST";
wReq.AllowAutoRedirect = false;
wReq.ContentType = "application/x-www-form-urlencoded";
string postData = "ctl00$cphContent$txtUserName=Username&ctl00$cphContent$txtPassword=Password";
byte[] dataBytes = UTF8Encoding.UTF8.GetBytes(postData);
wReq.ContentLength = dataBytes.Length;
using (Stream postStream = wReq.GetRequestStream())
{
postStream.Write(dataBytes, 0, dataBytes.Length);
}
HttpWebResponse wResp = wReq.GetResponse() as HttpWebResponse;
string pageSource;
wReq = WebRequest.Create(url2) as HttpWebRequest;
wReq.CookieContainer = new CookieContainer();
wReq.CookieContainer.Add(wResp.Cookies);
HttpWebResponse wResp2 = wReq.GetResponse() as HttpWebResponse;
using (StreamReader sr = new StreamReader(wResp2.GetResponseStream()))
{
pageSource = sr.ReadToEnd();
}
Everytime I look at pageSource it is the HTML for the login.aspx page. I must be missing something here. Maybe it's not taking the cookie, I don't know. One question I have aside from, why doesn't this work, is in the string postData = "". Are those suppose to be the name or id portion of the html tag? Any help on this is greatly appreciated as I am stumped and will have to find a different way. I would like to continue with the WebRequest and WebResponse instead of using WebBrowser. If I can't, oh well. Thanks again for any help!
What are you trying to do besides login? If its like QAing a site programically, i would suggest using selenium andcreate a c# app based off of that. If u want i can post a link to a base project for a selenium based project.
Don't necessarily view the page source, but look at the actual HTTP POST. Install a HTTP proxy such as Fiddler and then re-visit the page you are trying to emulate. Complete the HTTP POST request, and check out the results produced in the proxy. From there you'll be able to see the actual parameters, cookies, headers, etc. that are being passed and you can then attempt to replicate this in your code. It's often easy to miss something when simply viewing the HTML source but monitoring the network traffic is pretty straight forward.
Basically I want to take a URLString and break out all variables and their values and Post them to another page with a redirect to taht page.. How do i pull this off without having a form and actuall submitting etc...
This is what i got..
string url = "http://www.blah.com/xyz.aspx";
StringBuilder postData = new StringBuilder();
postData.Append("CustomerID=" + HttpUtility.UrlEncode("Hello Rico") + "&");
postData.Append("FirstName=" + HttpUtility.UrlEncode("HelloFirstName"));
//ETC for all Form Elements
// Now to Send Data.
StreamWriter writer = null;
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded";
request.ContentLength = postData.ToString().Length;
try
{
writer = new StreamWriter(request.GetRequestStream());
writer.Write(postData.ToString());
writer.Flush();
HttpWebResponse WebResp = (HttpWebResponse)request.GetResponse();
//Now, we read the response (the string), and output it.
Stream Answer = WebResp.GetResponseStream();
StreamReader _Answer = new StreamReader(Answer);
Response.Write(_Answer.ReadToEnd());
}
finally
{
if (writer != null)
writer.Close();
}
Since you cannot redirect and post at the same time, your only option is to render an html page with a form that is submitted on page load. Unless you can make the receiver accept the parameters in the url of course. If it is an asp.net page you want to "post" to, it should not really matter if you send data in the querystring, unless of course you are sending massive amounts of data.
Remember that redirecting, is the same as telling the browser to "go to this page instead". This information is in the http header that you return. Therefore, there is not a lot of overhead in rendering a form that auto submits.
I'm not sure this is the right strategy - have you considered either HttpModules for manipulating requests or Server.Transfer for internal redirection?
What is it that you need the page you're redirecting the user toward to do? Your best option is to use Session Variables to pass the data between pages on Redirection unless you care to preserve the form that was initially passed to the page.
You can use code such as the following to add the information to a session item.
Session.Item("CustomerID") = "CustomerID=" & ID.ToString
This is driving me nuts and I can't figure out where I am dropping the ball. I've followed a few examples found via the googlemonsta to no avail. Any pointer to where I goofed would be greatly apperciated.
var writer = new StringWriter();
param = "location=" + Server.UrlEncode(param);
byte[] paramStream = Encoding.ASCII.GetBytes(param + "¶m2=value");
var URL = "http://www.somesite.com";
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(URL);
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded";
request.UserAgent = "Mozilla/5.0 (Windows; U; Windows NT 6.0; sv-SE; rv:1.9.1b2) Gecko/20081201 Firefox/3.1b2";
request.ContentLength = paramStream.Length;
using( var stream = request.GetRequestStream())
{
stream.Write(paramStream, 0, paramStream.Length);
}
var response = request.GetResponse();
string result;
using (var sr = new StreamReader(response.GetResponseStream()))
{
result = sr.ReadToEnd();
}
Thanks!
EDIT: As far as I can tell its hitting the site (i'm getting html back) but the params aren't pushed over. I'm basically getting where the values would appear had it been successful. I've tried removing the first & but didnt get anywhere.
EDIT: Edited code to reflect changes.
Possibly get rid of the & from the start of the first parameter? Other than that it basically looks okay. (Check the parameter names in your real code - where you've got "paramater" in the sample it should almost certainly be "parameter" - but we don't know what your real code looks like or what the real site expects.)
Please give more information about what's actually happening. We know it doesn't work, but there are a lot of different possible failure modes :)
One extra thought occurs - you haven't specified the content length. I'm not sure whether this is filled in automatically by WebRequest. It would be worth using WireShark to check whether or not it's present in the outbound request.
Just as a general point of practice, you should dispose of the WebResponse, and you don't need to call Close if you've already got a using statement for the response stream:
string result;
using (WebResponse response = request.GetResponse())
{
using (var sr = new StreamReader(response.GetResponseStream()))
{
result = sr.ReadToEnd();
}
}
Are you sure you have all the values neccessary for the post? I once had a case where there was a hidden input field on the form that was something like:
<input name="action" type="hidden" id="action" value="login">
and I had to supply that as a param as:
&action=login
Make sure you're not missing anything from the form is what I'm saying...
EDIT: One more thing: I just looked at my code again where I've done this, and noticed that I also had this line in there:
request.ContentLength = bytes.Length;
Not sure if you need that, but I noticed that you weren't setting the length.
It is not something as simple as a carriage-return/new-line after the parameters is it? Looking at some docs on HTTP on the internets, you apparently need a blank line afterwards.
(I would suggest telneting to the web server manually and pasting your request.)