I took an example of filling in a form in C# from another question, but when I run the code I just get the page back again (as if i didn't submit anything). I tried manually going to the URL and added ?variable=value&variable2=value2 but that didn't seem to pre-populate the form, not sure if that is why this isn't working.
private void button2_Click(object sender, EventArgs e)
{
var encoding = new ASCIIEncoding();
var postData = "appid=001";
postData += ("&email=chris#test.com");
postData += ("&receipt=testing");
postData += ("&machineid=219481142226.1");
byte[] data = encoding.GetBytes(postData);
var myRequest =
(HttpWebRequest)WebRequest.Create("http://www.example.com/licensing/check.php");
myRequest.Method = "POST";
myRequest.ContentType = "application/x-www-form-urlencoded";
myRequest.ContentLength = data.Length;
var newStream = myRequest.GetRequestStream();
newStream.Write(data, 0, data.Length);
newStream.Close();
var response = myRequest.GetResponse();
var responseStream = response.GetResponseStream();
var responseReader = new StreamReader(responseStream);
label2.Text = responseReader.ReadToEnd();
}
To submit a form programmatically, the general idea is that you want to impersonate a browser. To do this, you'll need to find the right combination of URL, HTTP headers, and POST data to satisfy the server.
The easiest way to figure out what the server needs is to use Fiddler, FireBug, or another tool which lets you examine exactly what the browser is sending over the wire. Then you can experiment in your code by adding or removing headers, changing POST data, etc. until the server accepts the request.
Here's a few gotchas you may run into:
first, make sure you're submitting the form to the right target URL. Many forms don't post to themselves but will post to a different URL
next, the form might be checking for a session cookie or authentication cookie, meaning you'll need to make one request (to pick up the cookie) and then make a subsequent cookied request to submit the form.
the form may have hidden fields you forgot to fill in. use Fiddler or Firebug to look at the form fields submitted when you fill in the form manually in the browser, and make sure to include the same fields in your code
depending on the server implementation, you may need to encode the # character as %40
There may be other challenges too, but those above are the most likely. To see the full list, take a look at my answer to another screen-scraping question.
BTW, the code you're using to submit the form is much harder and verbose than needed. Instead you can use WebClient.UploadValues() and accomplish the same thing with less code and with the encoding done automatically for you. Like this:
NameValueCollection postData = new NameValueCollection();
postData.Add ("appid","001");
postData.Add ("email","chris#test.com");
postData.Add ("receipt","testing");
postData.Add ("machineid","219481142226.1");
postData.Add ("checkit","checkit");
WebClient wc = new WebClient();
byte[] results = wc.UploadValues (
"http://www.example.com/licensing/check.php",
postData);
label2.Text = Encoding.ASCII.GetString(results);
UPDATE:
Given our discussion in the comments, the problem you're running into is one of the causes I originally noted above:
the form might be checking for a session cookie or authentication
cookie, meaning you'll need to make one request (to pick up the
cookie) and then make a subsequent cookied request to submit the form.
On a server that uses cookies for session tracking or authentication, if a request shows up without a cookie attached, the server will usually redirect to the same URL. The redirect will contain a Set-Cookie header, meaning when the redirected URL is re-requested, it will have a cookie attached by the client. This approach breaks if the first request is a form POST, because the server and/or the client isn't handling redirection of the POST.
The fix is, as I originally described, make an initial GET request to pick up the cookie, then make your POST as a second request, passing back the cookie.
Like this:
using System;
public class CookieAwareWebClient : System.Net.WebClient
{
private System.Net.CookieContainer Cookies = new System.Net.CookieContainer();
protected override System.Net.WebRequest GetWebRequest(Uri address)
{
System.Net.WebRequest request = base.GetWebRequest(address);
if (request is System.Net.HttpWebRequest)
{
var hwr = request as System.Net.HttpWebRequest;
hwr.CookieContainer = Cookies;
}
return request;
}
}
class Program
{
static void Main(string[] args)
{
var postData = new System.Collections.Specialized.NameValueCollection();
postData.Add("appid", "001");
postData.Add("email", "chris#test.com");
postData.Add("receipt", "testing");
postData.Add("machineid", "219481142226.1");
postData.Add("checkit","checkit");
var wc = new CookieAwareWebClient();
string url = "http://www.example.com/licensing/check.php";
// visit the page once to get the cookie attached to this session.
// PHP will redirect the request to ensure that the cookie is attached
wc.DownloadString(url);
// now that we have a valid session cookie, upload the form data
byte[] results = wc.UploadValues(url, postData);
string text = System.Text.Encoding.ASCII.GetString(results);
Console.WriteLine(text);
}
}
Related
EDIT
This question was for a workaround. Getting a successful login by using HttpWebRequests. Not on how to use the api.
Question
I noticed in the API there was no way to get a usernames password.
This is what I have now as a test. I thought I could just get the "Wrong user/pass" response first and go from there. All I get is the pages source code.
Anyone have any pointers or advice?
I am definitively logging in. In Account Admin and Login History, it shows me logging in. But the server is not serving any useful response text for the login. And now, I locked myself out using wrong passwords to sort through the streamreader lol.
public string DoVerification(string email, string password)
{
var request = (HttpWebRequest)WebRequest.Create("https://app.smartsheet.com/b/home");
var postData = "loginEmail=" + email;
postData += "&loginPassword=" + password;
postData += "&action=login";
var data = Encoding.ASCII.GetBytes(postData);
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded";
request.ContentLength = data.Length;
using (var stream = request.GetRequestStream())
{
stream.Write(data, 0, data.Length);
}
var response = (HttpWebResponse)request.GetResponse();
return new StreamReader(response.GetResponseStream()).ReadToEnd();
}
private void btnLogin_Click(object sender, EventArgs e)
{
string response = DoVerification("test#test.com", "12345");
MessageBox.Show(response.ToString());
}
It's unclear what you are trying to achieve or why you expect it to work.
There is no way to retrieve a password through the API. That would be a bad idea.
You aren't actually using the API. API endpoints start with https://api.smartsheet.com/2.0 and are documented here: http://smartsheet-platform.github.io/api-docs/
After comparing both sources from the returned response.
I have these two js functions.
Failed attempt
function loggedFailures() {
logExternalGTMEvent({'event': 'app-login-failure','method': 'onsite','error': 'AUTH_NO_MATCHING_USER'}); return true
}
Successful attempt
function loggedFailures() {
return false
}
I just simply check for one or the other.
And for the record, putting the users password in the api is not a bad idea.... Smartsheets lets us delete any user through the API, so I don't see what it would matter.
Ok, I've been racking my brain on this one solo for too long. I've been unable to crack it even with hours spent on this and many other sites.
Essentially, I'm trying to strip some data from a webpage behind a LogIn page using WebRequest/Response. (I have gotten this to work using a WebBrowser control with some layered events which navigate to the different web pages but it's causing some problems when trying to refactor - not to mention it's been stated that using a hidden form to do the work is 'bad practice'.)
This is what I have:
string formParams = string.Format("j_username={0}&j_password={1}", userName, userPass);
string cookieHeader;
WebRequest request = WebRequest.Create(_signInPage);
request.ContentType = "text/plain";
request.Method = "POST";
byte[] bytes = Encoding.ASCII.GetBytes(formParams);
request.ContentLength = bytes.Length;
using (Stream os = request.GetRequestStream())
{
os.Write(bytes, 0, bytes.Length);
}
WebResponse response = request.GetResponse();
cookieHeader = response.Headers["Set-Cookie"];
WebRequest getRequest = WebRequest.Create(sessionHistoryPage);
getRequest.Method = "GET";
getRequest.Headers.Add("Cookie", cookieHeader);
WebResponse getResponse = getRequest.GetResponse();
try
{
using (StreamReader sr = new StreamReader(getResponse.GetResponseStream()))
{
textBox1.AppendText(sr.ReadToEnd());
}
}
catch (Exception ex)
{
MessageBox.Show(ex.Message);
throw;
}
So far, I'm able to get to the proper page from the first link but when I go to the second, it sends me back to the login page as if I didn't log in.
The problem may lie in cookies not getting captured correctly but I'm a novice so maybe I'm just doing it wrong. It captures the cookies sent back from the POST: JSESSIONID and S2V however, when we go to the "GET", using FireFox WebConsole, the browser shows that it sends JSESSIONID, S2V and a SPRING_SECURITY_REMEMBER_ME_COOKIE, which I believe is the cookie used when I click the "Remember Me" box on the login form.
I've tried many different ways of doing this using the resources of SO but I have yet to get to the page I need. So, for the sake of the hair I have left, I've decided to ask for help on good ole SO. (This is one of those things I don't want to let up on - stubborn like that sometimes)
If someone wants the actual address of the site I'm trying to log into, I'd be more than happy to send it to a couple people in a private message.
Code that I have to reflect a suggested answer by Wal:
var request = (HttpWebRequest)WebRequest.Create(sessionHistoryPage);
request.Credentials = new NetworkCredential(userName, userPass);
request.CookieContainer = new CookieContainer();
request.PreAuthenticate = true;
WebResponse getResponse = request.GetResponse();
try
{
using (StreamReader sr = new StreamReader(getResponse.GetResponseStream()))
{
textBox1.AppendText(sr.ReadToEnd());
}
}
catch (Exception ex)
{
MessageBox.Show(ex.Message);
throw;
}
This suggestion, at least the way I implemented it, didn't work.
As Krizz suggested, I changed the code to use CookieContainer and transferring the cookies from one request to the other however, the response just gives me back the original login page as if I didn't login.
Are there certain sites that just WILL NOT allow this type of behavior?
Final Solution
The final solution was proposed by Adrian Iftode where he stated that the website I'm trying to log in might not allow to have an authentication without a valid session so adding a GET to the beginning of the process allowed me to get that cookie.
Thanks for all your help guys!
I was doing some sort of cookie transfer for a website written with PHP.
Clearly you are passing the cookie, but maybe is like in that situation
var phpsessid = response.Headers["Set-Cookie"].Replace("; path=/", String.Empty);
The Set-Cookie header contains other related info about the cookie and possible other instructions for other cookies. I had one cookie with its info (Path), the session id which I needed to sent back to the server so the server would know that I am the same client which did the GET request.
The new request had to include this cookie
request.Headers["Cookie"] = phpsessid;
You already do this, but make sure the cookies you receive, you sent back to the server.
Considering session and authentication, there are two cookies, one for session, one for authentication and some servers/application might not allow to have an authentication without a valid session. What I want to say is that you might need to pass the session cookie too. So the steps would be:
Do first a GET request to obtain the session cookie.
Next do the POST request to authenticate and get the auth cookie.
Use both cookies to navigate to the protected pages.
Also check this question, it doesn't show the entire class, but the idea is to keep the CookieContainer in the same class, add the new cookies from POST/GET requests and assign them to the each new request, like #Krizz answered.
Try using CookieContainer which is a class to keep cookies context between several requests. You simply create an instance of it and assign it to each WebRequest.
Therefore, modifying your code:
string formParams = string.Format("j_username={0}&j_password={1}", userName, userPass);
string cookieHeader;
var cookies = new CookieContainer(); // added this line
var request = WebRequest.Create(_signInPage) as HttpWebRequest; // modified line
request.CookieContainer = cookies; // added this line
request.ContentType = "text/plain";
request.Method = "POST";
byte[] bytes = Encoding.ASCII.GetBytes(formParams);
request.ContentLength = bytes.Length;
using (Stream os = request.GetRequestStream())
{
os.Write(bytes, 0, bytes.Length);
}
request.GetResponse(); // removed some code here, no need to read response manually
var getRequest = WebRequest.Create(sessionHistoryPage) as HttpWebRequest; //modified line
getRequest.CookieContainer = cookies; // added this line
getRequest.Method = "GET";
WebResponse getResponse = getRequest.GetResponse();
try
{
using (StreamReader sr = new StreamReader(getResponse.GetResponseStream()))
{
textBox1.AppendText(sr.ReadToEnd());
}
}
catch (Exception ex)
{
MessageBox.Show(ex.Message);
throw;
}
I know this question has been asked quite a lot of times which is how I have got to where I am at with the code below however I just can't get it to work on the particular website I am trying to access. At the site I am trying to access I need to retrieve certain values from the page however things like price and availability only come up after logging in so I am trying to submit my login information and then go to the product page to get the information I need using HTML Agility Pack.
At the moment it seems to attempt the login however the website is either not accepting it or the cookies are not present on the next page load to actually keep me logged in.
If someone could help me with this I would be very grateful as I am not a programmer but have been assigned this task as part of a software installation.
protected void Button5_Click(object sender, System.EventArgs e)
{
string LOGIN_URL = "http://www.videor.com/quicklogin/1/0/0/0/index.html";
string SECRET_PAGE_URL = "http://www.videor.com/item/47/32/0/703/index.html?scriptMode=&CUSTOMERNO=xxx&USERNAME=xxx&activeTabId=0";
// have a cookie container ready to receive the forms auth cookie
CookieContainer cookies = new CookieContainer();
// first, request the login form to get the viewstate value
HttpWebRequest webRequest = WebRequest.Create(LOGIN_URL) as HttpWebRequest;
webRequest.CookieContainer = cookies;
StreamReader responseReader = new StreamReader(
webRequest.GetResponse().GetResponseStream()
);
string responseData = responseReader.ReadToEnd();
responseReader.Close();
string postData = "CUSTOMERNO=xxxx&USERNAME=xxxxx&PASSWORD=xxxxx";
// now post to the login form
webRequest = WebRequest.Create(LOGIN_URL) as HttpWebRequest;
webRequest.Method = "POST";
webRequest.ContentType = "application/x-www-form-urlencoded";
webRequest.CookieContainer = cookies;
// write the form values into the request message
StreamWriter requestWriter = new StreamWriter(webRequest.GetRequestStream());
requestWriter.Write(postData);
requestWriter.Close();
// we don't need the contents of the response, just the cookie it issues
webRequest.GetResponse().Close();
// now we can send out cookie along with a request for the protected page
webRequest = WebRequest.Create(SECRET_PAGE_URL) as HttpWebRequest;
webRequest.CookieContainer = cookies;
responseReader = new StreamReader(webRequest.GetResponse().GetResponseStream());
// and read the response
responseData = responseReader.ReadToEnd();
responseReader.Close();
Response.Write(responseData);
}
This isn't a direct answer since I'm not sure what's wrong with your code (from a cursory glance it looks ok), but another approach is to use browser automation using Selenium . The following code will actually load the page using Chrome (you can swap out Firefox or IE) and is simpler to code against. It also won't break if they add javascript or something.
var driver = new ChromeDriver();
driver.Navigate().GoToUrl(LOGON_URL);
driver.FindElement(By.Id("UserName")).SendKeys("myuser");
driver.FindElement(By.Id("Password")).SendKeys("mypassword");
driver.FindElement(By.TagName("Form")).Submit();
driver.Navigate().GoToUrl(SECRET_PAGE_URL);
// And now the html can be found as driver.PageSource. You can also look for
// different elements and get their inner text and stuff as well.
This code is for an outlook plugin. We're trying to POST to a page and are getting this error:
The remote server returned an error: (422) Unprocessable Entity.
The C# code is here:
webClient.Headers.Add("Content-Type", "application/x-www-form-urlencoded");
ASCIIEncoding asciiEncoding = new System.Text.ASCIIEncoding();
Byte[] postData = asciiEncoding.GetBytes("email=e2#email.com&password=hunter2");
char[] resultHTML = asciiEncoding.GetChars(webClient.UploadData("http://url", "POST", postData));
string convertedResultHTML = new string(resultHTML);
Any idea what could be causing this?
POST data must be encoded prior to be sent out on the wire as ASCII, if you are sending character not in the ASCII range. You should try something like:
Byte[] postData = asciiEncoding.GetBytes(HttpUtility.UrlEncode("email=e2#email.com&password=hunter2"));
Because of its limited functionality, I avoid using WebClient and use WebRequest instead. The code below:
does not expect an HTTP 100 status code to be returned,
creates a CookieContainer to store any cookies we pick up,
sets the Content Length header, and
UrlEncodes each value in the post data.
Give the following a try and see if it works for you.
System.Net.ServicePointManager.Expect100Continue = false;
System.Net.CookieContainer cookies = new System.Net.CookieContainer();
// this first request just ensures we have a session cookie, if one exists
System.Net.WebRequest req = System.Net.WebRequest.Create("http://localhost/test.aspx");
((System.Net.HttpWebRequest)req).CookieContainer = cookies;
req.GetResponse().Close();
// this request submits the data to the server
req = System.Net.WebRequest.Create("http://localhost/test.aspx");
req.ContentType = "application/x-www-form-urlencoded";
req.Method = "POST";
((System.Net.HttpWebRequest)req).CookieContainer = cookies;
string parms = string.Format("email={0}&password={1}",
System.Web.HttpUtility.UrlEncode("e2#email.com"),
System.Web.HttpUtility.UrlEncode("hunter2"));
byte[] bytes = System.Text.Encoding.ASCII.GetBytes(parms);
req.ContentLength = bytes.Length;
// perform the POST
using (System.IO.Stream os = req.GetRequestStream())
{
os.Write(bytes, 0, bytes.Length);
}
// read the response
string response;
using (System.Net.WebResponse resp = req.GetResponse())
{
if (resp == null) return;
using (System.IO.StreamReader sr = new System.IO.StreamReader(resp.GetResponseStream()))
{
response = sr.ReadToEnd().Trim();
}
}
// the variable response holds the results of the request...
Credits: Hanselman, Simon (SO Question)
This is the RoR application telling you that you have not formed a request that it can handle; the destination script exists (otherwise you'd see a 404), the request is being handled (otherwise you'd get a 400 error) and it's been encoded correctly (or you'd get a 415 error) but the actual instruction can't be carried out.
Looking at it, you seem to be loading some email information. The RoR application could be telling you that the username and password is wrong, or that the user doesn't exist, or something else. It's up to the RoR application itself.
I think the code itself is good; it's just that the app at the other end isn't happy about doing what you ask it. Are you missing something else in the request information, like a command? (eg command=getnetemails&email=e2#email.com&password=hunter2) Are you sure the email/password combination you are passing is good?
see here for more on the 422 error.
Add the below line above your code.
System.Net.ServicePointManager.Expect100Continue = false;
Are you trying to access an authentication required page?
it was solved by returning xml instead of just unstructured text on the RoR side
I'm currently writing some tests so that I may improve my skills with the Internet interaction through Windows Forms. One of those tests is to find a postal code which should be returned by Canada Post website.
My default URL setting is set to: http://www.canadapost.ca/cpotools/apps/fpc/personal/findByCity?execution=e4s1
The required form fields are: streetNumber, streetName, city, province
The contentType is "application/x-www-form-enclosed"
EDIT: Please consider the value "application/x-www-form-encoded" instead of point 3 value as the contentType. (Thanks EricLaw-MSFT!)
The result I get is not the result expected. I get the HTML source code of the page where I could manually enter the information to find the postal code, but not the HTML source code with the found postal code. Any idea of what I'm doing wrong?
Shall I consider going the XML way? Is it first of all possible to search on Canada Post anonymously?
Here's a code sample for better description:
public static string FindPostalCode(ICanadadianAddress address) {
var postData = string.Concat(string.Format("&streetNumber={0}", address.StreetNumber)
, string.Format("&streetName={0}", address.StreetName)
, string.Format("&city={0}", address.City)
, string.Format("&province={0}", address.Province));
var encoding = new ASCIIEncoding();
byte[] postDataBytes = encoding.GetBytes(postData);
request = (HttpWebRequest)WebRequest.Create(DefaultUrlSettings);
request.ImpersonationLevel = System.Security.Principal.TokenImpersonationLevel.Anonymous;
request.Container = new CookieContainer();
request.Timeout = 10000;
request.ContentType = contentType;
request.ContentLength = postDataBytes.LongLength;
request.Method = #"post";
var senderStream = new StreamWriter(request.GetRequestStream());
senderStream.Write(postDataBytes, 0, postDataBytes.Length);
senderStream.Close();
string htmlResponse = new StreamReader(request.GetResponse().GetResponseStream()).ReadToEnd();
return processedResult(htmlResponse); // Processing the HTML source code parsing, etc.
}
I seem stuck in a bottle neck in my point of view. I find no way out to the desired result.
EDIT: There seems to have to parameters as for the ContentType of this site. Let me explain.
There's one with the "meta"-variables which stipulates the following:
meta http-equiv="Content-Type" content="application/xhtml+xml,
text/xml, text/html; charset=utf-8"
And another one later down the code that is read as:
form id="fpcByAdvancedSearch:fpcSearch" name="fpcByAdvancedSearch:fpcSearch" method="post" action="/cpotools/apps/fpc/personal/findByCity?execution=e1s1" enctype="application/x-www-form-urlencoded"
My question is the following: With which one do I have to stick?
Let me guess, the first ContentType is to be considered as the second is only for another request to a function or so when the data is posted?
EDIT: As per request, the closer to the solution I am is listed under this question:
WebRequest: How to find a postal code using a WebRequest against this ContentType=”application/xhtml+xml, text/xml, text/html; charset=utf-8”?
Thanks for any help! :-)
I'm trying to see a reason why you are not using the WebClient class:-
var fields = new NameValueCollection();
fields.Add("streetnumber", address.StreetNumber);
fields.Add("streetname", address.StreetName);
fields.Add("city", address.City);
fields.Add("province", address.Province);
var wc = new WebClient();
byte[] resultData = wc.UploadValues(url, fields);
string result = Encoding.Default.GetString(resultData);
You might want to check the encoding used by the server when sending the results, if it uses UTF-8 change the last line to:-
string result = Encoding.UTF8.GetString(resultData);
Some issues I spot in your orginal code:-
The first field is prefixed with &, that shouldn't be there.
You need call use Uri.EscapeDataString on each field value.
You are attempting to construct a memory stream around the result of GetRequestStream, I can't see what that would achieve even if MemoryStream had such a constructor but it doesn't anyway. Just write directly to the stream returned by GetRequestStream
If you have already done so get yourself a copy of fiddler so you can observe what occurs when a standard form requests the data sucessfully and what your code is doing.
Edit: If you have evidence that the lack of a cookie container is what causes WebClient not to work then you could try this approach:-
public class MyWebClient : WebClient
{
protected override WebRequest GetWebRequest (Uri address)
{
WebRequest request = (WebRequest) base.GetWebRequest (address);
request.Container = new CookieContainer();
return request;
}
}
Now use my code above to but instead od instancing WebClient instance MyWebClient instead.
HTTPWebRequest will return the content of the URLrequested. If it is a HTML page, it will return HTML markup. If the page content is XML, then it will return XML markup.
It sounds like you need is a Webservice. I would see if that site has any webserivces available to process that type of request. If they do, then it will return XML, JSON markup in response to your query. Otherwise you are left to parsing the output of the request.