I need to visit a URL, find a specific text box in said page - fill it with data and then submit a form.
How can I accomplish this in C#?
P.S. Innocent intentions.
You'd be best looking at the WebRequest class (System.Net).
You'll want to look at the POST Method to post a form (click the submit button with the required fields completed).
Example:
// Create a request using a URL that can receive a post.
WebRequest request = WebRequest.Create ("http://www.contoso.com/PostAccepter.aspx ");
// Set the Method property of the request to POST.
request.Method = "POST";
// Create POST data and convert it to a byte array.
string postData = "This is a test that posts this string to a Web server.";
byte[] byteArray = Encoding.UTF8.GetBytes (postData);
There is a nice tutorial and lots of information on MSDN here. (Continuation of above source code)
if you visit the page in a browser and do the stuff you need manually with fiddler2 installed and activated you can see which requests get sent to the webserver.
All you need to do is to replicate those requests (form posts) using for example the WebRequest class or the WebClient class in the .net framework.
WatiN is also an alternative.
Related
Hello i making a simple httpwebrequest and then i read (StreamReader) the response and just want to get the html page of website,but i get only one laber(only one element of the page) in the browser all fine(i see all page) but when i try to set cookies to Deny\disable i also in the browser get this label(only one element of the page) and all is disappear.Sow i getting opinion if after i disabled cookies in browser i get the same page(like in code) that mean my HttpWebRequest is have settings cookies=deny/disable.
You can go to https://www.bbvanetcash.com/local_kyop/KYOPSolicitarCredenciales.html and disable cookies with F12 and you will see the difrance and also this page with one label.
Sow this my code any ideas what i need to change here?
HttpWebRequest myHttpWebRequest = (HttpWebRequest)WebRequest.Create("https://www.bbvanetcash.com/local_kyop/KYOPSolicitarCredenciales.html");
HttpWebResponse myHttpWebResponse = (HttpWebResponse)myHttpWebRequest.GetResponse();
Stream streamResponseLogin = myHttpWebResponse.GetResponseStream();
StreamReader streamReadLogin = new StreamReader(streamResponseLogin);
LoginInfo = streamReadLogin.ReadToEnd();
Your code is receiving complete page content, but it cannot receive the dynamic contents. This is happening because the page you are trying to access relies on Cookies for maintaining session as well as JavaScript (it is using jQuery) for loading dynamic contents and providing rich user experience.
To successfully receive the whole page, your code must
support retrieving, storing and sending cookie objects across various HttpRequest and HttpResponse.
be able to execute JavaScript code to load the dynamic contents/markup of the page
To test 'if your code is receiving proper values or not' visit the site Web Sniffer and put your URL there.
As you can try on web-sniffer site, for www.google.com, the response you are getting is a redirect instruction.... that means, even to access the Google's home page, your code must understand HTTP status messages (302 there).
I have a page in .Net that currently does some processing of info when a button is clicked. It posts back, updates some info and then redirects the user onwards.
What I want to do now is for this same button, when it's clicked, the info is updated but instead of a redirect it does a POST to another site. The reason being that this other site needs to read a bunch of data from the form I submit.
So, I know about the PostBackUrl property but that will stop the processing of the data that I need done.
So is there another way for me to be able to somehow combine both a postback that then becomes as POST to another site?
Or alternatively some way for me to be able to do the updates I need and then do a POST?
The suggested solutions all kind-of worked but the only one that actually did exactly what I needed it to was this one:
Link to SO answer
The reason the other answers above didn't work is that I was POSTing to a payment gateway and for whatever reason their system thought there was a problem with various missing fields in all solutions except the one I linked to. No idea why, I don't have access to their systems to know what they were actually doing.
In any case, thanks for all answers but have a look at the linked one if you're running into a similar issue.
if PostBack isn't absolutely needed, you can actually send them in the request query itself.
You can do a POST from codebehind, you can find details in this answer Redirect to another page using Post method from Code behind
If I got your question right I think you will need to get all the form data from Request.Form and make a HttpWebRequest to the other site:
string url = "http://anothersite.com/";
// create post data
StringBuilder postDataBuilder = new StringBuilder();
foreach (var key in this.Request.Form.AllKeys)
{
postDataBuilder.AppendFormat("{0}={1}&", this.Request.Form[key]);
}
string postData = postDataBuilder.ToString();
// create the web request for the POST
HttpWebRequest webRequest = (HttpWebRequest)WebRequest.Create(url);
webRequest.Method = "POST";
webRequest.ContentType = "application/x-www-form-urlencoded";
webRequest.ContentLength = postData.Length;
// add the post data
using (StreamWriter requestStream = new StreamWriter(webRequest.GetRequestStream()))
{
requestStream.Write(postData);
}
Hope this helps!
Similar threads may be:
How do I programmatically send information to a web service in C# with .Net?
Maintain button submit value when using link to submit a form
Looking how to send multiple values with an input type=submit form
How send a form with Javascript when input name is "submit"?
How do you submit a form with javascript with an < input type="button">
...All of the above seem to almost answer the question...but I'm totally mystified. I think something along the lines of "Net.Post(something)" is how it is done...but I am not sure.
I am currently using F#, and I have figured out how to parse links. I also figured out how to catch the various search bars and submission buttons with regexes.
I would like to use my code to search for something on a search engine by specifically:
First, obtaining the HTML
Second, Scraping the HTML for the various buttons, text bars, and links
Third, use some unknown method/device/tool/function to send a search thread to a text box...
Fourth, Simulate an actual mouse click on the submit "button" that appears on the website...
...
Then, upon reciept of the server's response, pull the HTML from the next site.
Here is my link code as it stands:
type Url(x:string)=
member this.Tostring = sprintf "%A" x
member this.Request = System.Net.WebRequest.Create(x)
member this.Response = this.Request.GetResponse()
member this.Stream = this.Response.GetResponseStream()
member this.Reader = new System.IO.StreamReader(this.Stream)
member this.Html = this.Reader.ReadToEnd()
let linkex = "href=\s*\"[^\"h]*(http://[^&\"]*)\""
let getLinks (txt:string) = [for m in Regex.Matches(txt,linkex)
-> m.Groups.Item(1).Value ]
let collectLinks (url:Url) = url.html
|> getLinks
... I know how to grab the search box strings and whatnot...the question is, when I go to, say, google.com and grab the search bar the same way I grab the links, update the value field with my search string, how do I then submit the updated search bar to google's server?
Secondly, how do I do it if I want to update it and then simulate a mouse click?
In other words, I want to interact with websites the way the user interacts with websites.
There are essentially two options - some web pages accept data using HTTP GET (which means that the data is sent as part of the URL) and other use HTTP POST (which means that the data is sent as the body of the request).
If you're using i.e. Google, then you can use HTTP GET and put the query string in the URL. For example, you can just download a web page with the following URL: https://www.google.com/search?q=hello. So, all you need to do is to generate the URL as follows:
let search = sprintf "http://www.google.com/search?q=%s"
If you want to send HTTP POST request from F#, then you need to create a request with body that contains the form values in an encoded form. This can be written as follows:
open System.Text
open System.IO
open System.Net
// URL of a simple page that takes two HTTP POST parameters. See the
// form that submits there: http://www.snee.com/xml/crud/posttest.html
let url = "http://www.snee.com/xml/crud/posttest.cgi"
// Create & configure HTTP web request
let req = HttpWebRequest.Create(url) :?> HttpWebRequest
req.ProtocolVersion <- HttpVersion.Version10
req.Method <- "POST"
// Encode body with POST data as array of bytes
let postBytes = Encoding.ASCII.GetBytes("fname=Tomas&lname=Petricek")
req.ContentType <- "application/x-www-form-urlencoded";
req.ContentLength <- int64 postBytes.Length
// Write data to the request
let reqStream = req.GetRequestStream()
reqStream.Write(postBytes, 0, postBytes.Length);
reqStream.Close()
// Obtain response and download the resulting page
// (The sample contains the first & last name from POST data)
let resp = req.GetResponse()
let stream = resp.GetResponseStream()
let reader = new StreamReader(stream)
let html = reader.ReadToEnd()
Aside, your use of type with member for every step is a bit weird. Members are re-executed each time you access them, so the code you wrote is pretty non-deterministic. You should use let binding instead.
Am trying to post using HTTPWebrequest and this is the response i keep getting back:
you must use a browser that supports and has JavaScript enabled
This is my post code:
HttpWebRequest myRequest = null;
myRequest = (HttpWebRequest)HttpWebRequest.Create(submitURL);
myRequest.Headers.Add("Accept-Language", "en-US");
myRequest.Accept = "image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, application/x-shockwave-flash, application/xaml+xml, application/vnd.ms-xpsdocument, application/x-ms-xbap, application/x-ms-application, application/vnd.ms-excel, application/vnd.ms-powerpoint, application/msword, */*";
myRequest.Method = WebRequestMethods.Http.Post;
myRequest.Headers.Add("Accept-Language", "en-US");
myRequest.Accept = "*/*, text/xml";
myRequest.ContentType = "application/x-www-form-urlencoded" + "\n" + "\r";
myRequest.CookieContainer = cookieContainer;
myRequest.Headers.Add("UA-CPU", "x86");
myRequest.Headers.Add("Accept-Encoding", "gzip, deflate");
//cPostData section removed as submitting to SO
myRequest.ContentLength = cPostData.Length;
myRequest.ServicePoint.Expect100Continue = false;
StreamWriter streamWriter = new System.IO.StreamWriter(myRequest.GetRequestStream());
streamWriter.Write(cPostData);
streamWriter.Close();
HttpWebResponse httpWebResponse = (HttpWebResponse)myRequest.GetResponse();
StreamReader streamReader = new System.IO.StreamReader(httpWebResponse.GetResponseStream());
string stringResult = streamReader.ReadToEnd();
streamReader.Close();
how do i avoid getting this error?
It is difficult to say what the exact problem is because the server that is receiving you request doesn't think it is valid.
Perhaps the first thing to try would be to set the UserAgent property on your HttpWebRequest to some valid browser's user agent string as the server may be using this value to determine whether or not to serve the page.
This doesn't have anything to do with your code - the web server code has something that detects or relies on Javascript. Most likely a piece of Javascript on the page fills out (or modifies prior to posting) some hidden form field(s).
The solution to this is entirely dependent on what the web server is expecting to happen with that form data.
This is a layman's answer, and not a 100% technically accurate description of the httpWebRequest object, and meant that way bcause of the amount of time it would take to post it. The first part of this answer is to clarify the final sentence.
The httpWebRequest object basically acts as a browser that is interacting with web pages. It's a very simple browser, with no UI. it's designed basically to be able to post to and read from web pages. As such, it does not support a variety of features normally found in a browser these days, such as JavaScript.
The page you are attempting to post to requires javascript, which the httpWebRequest object does not support. If you have no control over the page that the WebRequst object is posting to, then you'll have to find another wat to post to it. If you own or control the page, you will need to modify the page to strip out items that require javascript (such as Ajax features, etc).
Added
I purposely didn't add anything about specifying a user-agent to try to trick the web server into thinking the httpWebRequest object supports JavaScript. because it is likely that the page really needs to have JavaScript enabled in order for the page to be displayed properly. However, a lot of my assumptions prove wrong, so I would agree with #Andrew Hare and say it's worth a try.
I have a project at work the requires me to be able to enter information into a web page, read the next page I get redirected to and then take further action. A simplified real-world example would be something like going to google.com, entering "Coding tricks" as search criteria, and reading the resulting page.
Small coding examples like the ones linked to at http://www.csharp-station.com/HowTo/HttpWebFetch.aspx tell how to read a web page, but not how to interact with it by submitting information into a form and continuing on to the next page.
For the record, I'm not building a malicious and/or spam related product.
So how do I go read web pages that require a few steps of normal browsing to reach first?
You can programmatically create an Http request and retrieve the response:
string uri = "http://www.google.com/search";
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uri);
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded";
// encode the data to POST:
string postData = "q=searchterm&hl=en";
byte[] encodedData = new ASCIIEncoding().GetBytes(postData);
request.ContentLength = encodedData.Length;
Stream requestStream = request.GetRequestStream();
requestStream.Write(encodedData, 0, encodedData.Length);
// send the request and get the response
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
// Do something with the response stream. As an example, we'll
// stream the response to the console via a 256 character buffer
using (StreamReader reader = new StreamReader(response.GetResponseStream()))
{
Char[] buffer = new Char[256];
int count = reader.Read(buffer, 0, 256);
while (count > 0)
{
Console.WriteLine(new String(buffer, 0, count));
count = reader.Read(buffer, 0, 256);
}
} // reader is disposed here
} // response is disposed here
Of course, this code will return an error since Google uses GET, not POST, for search queries.
This method will work if you are dealing with specific web pages, as the URLs and POST data are all basically hard-coded. If you needed something that was a little more dynamic, you'd have to:
Capture the page
Strip out the form
Create a POST string based on the form fields
FWIW, I think something like Perl or Python might be better suited to that sort of task.
edit: x-www-form-urlencoded
You might try Selenium. Record the actions in Firefox using Selenium IDE, save the script in C# format, then play them back using the Selenium RC C# wrapper. As others have mentioned you could also use System.Net.HttpWebRequest or System.Net.WebClient. If this is a desktop application see also System.Windows.Forms.WebBrowser.
Addendum: Similar to Selenium IDE and Selenium RC, which are Java-based, WatiN Test Recorder and WatiN are .NET-based.
What you need to do is keep retrieving and analyzing the html source for each page in the chain. For each page, you need to figure out what the form submission will look like and send a request that will match that to get the next page in the chain.
What I do is build a custom class the wraps System.Net.HttpWebRequest/HttpWebResponse, so retrieving pages is as simple as using System.Net.WebClient. However, my custom class also keeps the same cookie container across requests and makes it a little easier to send post data, customize the user agent, etc.
Depending on how the website works you can either manipulate the url to perform what you want. e.g to search for the word "beatles" you could just open a request to google.com?q=beetles and then just read the results.
Alternatively if the website does not use querystring values (url) to process page actions then you will need to work on a webrequest which posts the required values to the website instead. Search in Google for working with WebRequest and webresponse.