We Keep Coding

Login session not transferring to new webpage using WebRequest/Response?

Ok, I've been racking my brain on this one solo for too long. I've been unable to crack it even with hours spent on this and many other sites.
Essentially, I'm trying to strip some data from a webpage behind a LogIn page using WebRequest/Response. (I have gotten this to work using a WebBrowser control with some layered events which navigate to the different web pages but it's causing some problems when trying to refactor - not to mention it's been stated that using a hidden form to do the work is 'bad practice'.)
This is what I have:
string formParams = string.Format("j_username={0}&j_password={1}", userName, userPass);
string cookieHeader;
WebRequest request = WebRequest.Create(_signInPage);
request.ContentType = "text/plain";
request.Method = "POST";
byte[] bytes = Encoding.ASCII.GetBytes(formParams);
request.ContentLength = bytes.Length;
using (Stream os = request.GetRequestStream())
{
os.Write(bytes, 0, bytes.Length);
}
WebResponse response = request.GetResponse();
cookieHeader = response.Headers["Set-Cookie"];
WebRequest getRequest = WebRequest.Create(sessionHistoryPage);
getRequest.Method = "GET";
getRequest.Headers.Add("Cookie", cookieHeader);
WebResponse getResponse = getRequest.GetResponse();
try
{
using (StreamReader sr = new StreamReader(getResponse.GetResponseStream()))
{
textBox1.AppendText(sr.ReadToEnd());
}
}
catch (Exception ex)
{
MessageBox.Show(ex.Message);
throw;
}
So far, I'm able to get to the proper page from the first link but when I go to the second, it sends me back to the login page as if I didn't log in.
The problem may lie in cookies not getting captured correctly but I'm a novice so maybe I'm just doing it wrong. It captures the cookies sent back from the POST: JSESSIONID and S2V however, when we go to the "GET", using FireFox WebConsole, the browser shows that it sends JSESSIONID, S2V and a SPRING_SECURITY_REMEMBER_ME_COOKIE, which I believe is the cookie used when I click the "Remember Me" box on the login form.
I've tried many different ways of doing this using the resources of SO but I have yet to get to the page I need. So, for the sake of the hair I have left, I've decided to ask for help on good ole SO. (This is one of those things I don't want to let up on - stubborn like that sometimes)
If someone wants the actual address of the site I'm trying to log into, I'd be more than happy to send it to a couple people in a private message.
Code that I have to reflect a suggested answer by Wal:
var request = (HttpWebRequest)WebRequest.Create(sessionHistoryPage);
request.Credentials = new NetworkCredential(userName, userPass);
request.CookieContainer = new CookieContainer();
request.PreAuthenticate = true;
WebResponse getResponse = request.GetResponse();
try
{
using (StreamReader sr = new StreamReader(getResponse.GetResponseStream()))
{
textBox1.AppendText(sr.ReadToEnd());
}
}
catch (Exception ex)
{
MessageBox.Show(ex.Message);
throw;
}
This suggestion, at least the way I implemented it, didn't work.
As Krizz suggested, I changed the code to use CookieContainer and transferring the cookies from one request to the other however, the response just gives me back the original login page as if I didn't login.
Are there certain sites that just WILL NOT allow this type of behavior?
Final Solution
The final solution was proposed by Adrian Iftode where he stated that the website I'm trying to log in might not allow to have an authentication without a valid session so adding a GET to the beginning of the process allowed me to get that cookie.
Thanks for all your help guys!

I was doing some sort of cookie transfer for a website written with PHP.
Clearly you are passing the cookie, but maybe is like in that situation
var phpsessid = response.Headers["Set-Cookie"].Replace("; path=/", String.Empty);
The Set-Cookie header contains other related info about the cookie and possible other instructions for other cookies. I had one cookie with its info (Path), the session id which I needed to sent back to the server so the server would know that I am the same client which did the GET request.
The new request had to include this cookie
request.Headers["Cookie"] = phpsessid;
You already do this, but make sure the cookies you receive, you sent back to the server.
Considering session and authentication, there are two cookies, one for session, one for authentication and some servers/application might not allow to have an authentication without a valid session. What I want to say is that you might need to pass the session cookie too. So the steps would be:
Do first a GET request to obtain the session cookie.
Next do the POST request to authenticate and get the auth cookie.
Use both cookies to navigate to the protected pages.
Also check this question, it doesn't show the entire class, but the idea is to keep the CookieContainer in the same class, add the new cookies from POST/GET requests and assign them to the each new request, like #Krizz answered.

Try using CookieContainer which is a class to keep cookies context between several requests. You simply create an instance of it and assign it to each WebRequest.
Therefore, modifying your code:
string formParams = string.Format("j_username={0}&j_password={1}", userName, userPass);
string cookieHeader;
var cookies = new CookieContainer(); // added this line
var request = WebRequest.Create(_signInPage) as HttpWebRequest; // modified line
request.CookieContainer = cookies; // added this line
request.ContentType = "text/plain";
request.Method = "POST";
byte[] bytes = Encoding.ASCII.GetBytes(formParams);
request.ContentLength = bytes.Length;
using (Stream os = request.GetRequestStream())
{
os.Write(bytes, 0, bytes.Length);
}
request.GetResponse(); // removed some code here, no need to read response manually
var getRequest = WebRequest.Create(sessionHistoryPage) as HttpWebRequest; //modified line
getRequest.CookieContainer = cookies; // added this line
getRequest.Method = "GET";
WebResponse getResponse = getRequest.GetResponse();
try
{
using (StreamReader sr = new StreamReader(getResponse.GetResponseStream()))
{
textBox1.AppendText(sr.ReadToEnd());
}
}
catch (Exception ex)
{
MessageBox.Show(ex.Message);
throw;
}

Fill Form C# & Post Error

I took an example of filling in a form in C# from another question, but when I run the code I just get the page back again (as if i didn't submit anything). I tried manually going to the URL and added ?variable=value&variable2=value2 but that didn't seem to pre-populate the form, not sure if that is why this isn't working.
private void button2_Click(object sender, EventArgs e)
{
var encoding = new ASCIIEncoding();
var postData = "appid=001";
postData += ("&email=chris#test.com");
postData += ("&receipt=testing");
postData += ("&machineid=219481142226.1");
byte[] data = encoding.GetBytes(postData);
var myRequest =
(HttpWebRequest)WebRequest.Create("http://www.example.com/licensing/check.php");
myRequest.Method = "POST";
myRequest.ContentType = "application/x-www-form-urlencoded";
myRequest.ContentLength = data.Length;
var newStream = myRequest.GetRequestStream();
newStream.Write(data, 0, data.Length);
newStream.Close();
var response = myRequest.GetResponse();
var responseStream = response.GetResponseStream();
var responseReader = new StreamReader(responseStream);
label2.Text = responseReader.ReadToEnd();
}

To submit a form programmatically, the general idea is that you want to impersonate a browser. To do this, you'll need to find the right combination of URL, HTTP headers, and POST data to satisfy the server.
The easiest way to figure out what the server needs is to use Fiddler, FireBug, or another tool which lets you examine exactly what the browser is sending over the wire. Then you can experiment in your code by adding or removing headers, changing POST data, etc. until the server accepts the request.
Here's a few gotchas you may run into:
first, make sure you're submitting the form to the right target URL. Many forms don't post to themselves but will post to a different URL
next, the form might be checking for a session cookie or authentication cookie, meaning you'll need to make one request (to pick up the cookie) and then make a subsequent cookied request to submit the form.
the form may have hidden fields you forgot to fill in. use Fiddler or Firebug to look at the form fields submitted when you fill in the form manually in the browser, and make sure to include the same fields in your code
depending on the server implementation, you may need to encode the # character as %40
There may be other challenges too, but those above are the most likely. To see the full list, take a look at my answer to another screen-scraping question.
BTW, the code you're using to submit the form is much harder and verbose than needed. Instead you can use WebClient.UploadValues() and accomplish the same thing with less code and with the encoding done automatically for you. Like this:
NameValueCollection postData = new NameValueCollection();
postData.Add ("appid","001");
postData.Add ("email","chris#test.com");
postData.Add ("receipt","testing");
postData.Add ("machineid","219481142226.1");
postData.Add ("checkit","checkit");
WebClient wc = new WebClient();
byte[] results = wc.UploadValues (
"http://www.example.com/licensing/check.php",
postData);
label2.Text = Encoding.ASCII.GetString(results);
UPDATE:
Given our discussion in the comments, the problem you're running into is one of the causes I originally noted above:
the form might be checking for a session cookie or authentication
cookie, meaning you'll need to make one request (to pick up the
cookie) and then make a subsequent cookied request to submit the form.
On a server that uses cookies for session tracking or authentication, if a request shows up without a cookie attached, the server will usually redirect to the same URL. The redirect will contain a Set-Cookie header, meaning when the redirected URL is re-requested, it will have a cookie attached by the client. This approach breaks if the first request is a form POST, because the server and/or the client isn't handling redirection of the POST.
The fix is, as I originally described, make an initial GET request to pick up the cookie, then make your POST as a second request, passing back the cookie.
Like this:
using System;
public class CookieAwareWebClient : System.Net.WebClient
{
private System.Net.CookieContainer Cookies = new System.Net.CookieContainer();
protected override System.Net.WebRequest GetWebRequest(Uri address)
{
System.Net.WebRequest request = base.GetWebRequest(address);
if (request is System.Net.HttpWebRequest)
{
var hwr = request as System.Net.HttpWebRequest;
hwr.CookieContainer = Cookies;
}
return request;
}
}
class Program
{
static void Main(string[] args)
{
var postData = new System.Collections.Specialized.NameValueCollection();
postData.Add("appid", "001");
postData.Add("email", "chris#test.com");
postData.Add("receipt", "testing");
postData.Add("machineid", "219481142226.1");
postData.Add("checkit","checkit");
var wc = new CookieAwareWebClient();
string url = "http://www.example.com/licensing/check.php";
// visit the page once to get the cookie attached to this session.
// PHP will redirect the request to ensure that the cookie is attached
wc.DownloadString(url);
// now that we have a valid session cookie, upload the form data
byte[] results = wc.UploadValues(url, postData);
string text = System.Text.Encoding.ASCII.GetString(results);
Console.WriteLine(text);
}
}

.NET POST to PHP Page

I am trying to implement a very basic system from my C# .NET application that sends the IP address of the machine to authenticate.php on my web server. The php page will check this IP address against a database and either respond back with "yes" or "no".
It has been a long time since I worked with PHP, and I am a little bit confused. Here is what my .NET function looks like.
public static bool IsAuthenticated()
{
string sData = getPublicIP();
Uri uri = new Uri("http://www.mysite.com/authenticate.php");
if (uri.Scheme == Uri.UriSchemeHttp)
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uri);
request.Method = WebRequestMethods.Http.Post;
request.ContentLength = sData.Length;
request.ContentType = "application/x-www-form-urlencoded";
// POST the data to the authentication page
StreamWriter writer = new StreamWriter(request.GetRequestStream());
writer.Write(sData);
writer.Close();
// Retrieve response from authentication page
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
StreamReader reader = new StreamReader(response.GetResponseStream());
string sResponse = reader.ReadToEnd();
response.Close();
if (sResponse == "yes")
{
Console.WriteLine("Authentication was Successful.");
return true;
}
else
{
Console.WriteLine("Authentication Failed!");
return false;
}
}
}
So would the POST variable be $_POST['sData']; and how do I respond back to my application with the result?

Assuming the value of sData is (say) "10.1.1.1" then you're currently not posting proper form data in the first place. The name of the variable isn't part of the text written by
writer.Write(sData);
You need to do something like:
string postData = "ipaddress=" + sData;
and then use the ipaddress form parameter within your PHP.
Note also that you should be giving the binary content length, which may not be the same as the string length in characters. Of course it's okay if the string here is entirely ASCII, which I'd expect if it's an IP address... but it's worth bearing in mind for other uses. (Likewise you would normally need to bear in mind any characters which need special encoding.)
Also note that it would be better to use using statements for the StreamWriter, HttpResponse etc, to make sure that everything gets closed even if an exception is thrown.

getting this error: "The remote server returned an error: (422) Unprocessable Entity." when doing post from C# to RoR

This code is for an outlook plugin. We're trying to POST to a page and are getting this error:
The remote server returned an error: (422) Unprocessable Entity.
The C# code is here:
webClient.Headers.Add("Content-Type", "application/x-www-form-urlencoded");
ASCIIEncoding asciiEncoding = new System.Text.ASCIIEncoding();
Byte[] postData = asciiEncoding.GetBytes("email=e2#email.com&password=hunter2");
char[] resultHTML = asciiEncoding.GetChars(webClient.UploadData("http://url", "POST", postData));
string convertedResultHTML = new string(resultHTML);
Any idea what could be causing this?

POST data must be encoded prior to be sent out on the wire as ASCII, if you are sending character not in the ASCII range. You should try something like:
Byte[] postData = asciiEncoding.GetBytes(HttpUtility.UrlEncode("email=e2#email.com&password=hunter2"));

Because of its limited functionality, I avoid using WebClient and use WebRequest instead. The code below:
does not expect an HTTP 100 status code to be returned,
creates a CookieContainer to store any cookies we pick up,
sets the Content Length header, and
UrlEncodes each value in the post data.
Give the following a try and see if it works for you.
System.Net.ServicePointManager.Expect100Continue = false;
System.Net.CookieContainer cookies = new System.Net.CookieContainer();
// this first request just ensures we have a session cookie, if one exists
System.Net.WebRequest req = System.Net.WebRequest.Create("http://localhost/test.aspx");
((System.Net.HttpWebRequest)req).CookieContainer = cookies;
req.GetResponse().Close();
// this request submits the data to the server
req = System.Net.WebRequest.Create("http://localhost/test.aspx");
req.ContentType = "application/x-www-form-urlencoded";
req.Method = "POST";
((System.Net.HttpWebRequest)req).CookieContainer = cookies;
string parms = string.Format("email={0}&password={1}",
System.Web.HttpUtility.UrlEncode("e2#email.com"),
System.Web.HttpUtility.UrlEncode("hunter2"));
byte[] bytes = System.Text.Encoding.ASCII.GetBytes(parms);
req.ContentLength = bytes.Length;
// perform the POST
using (System.IO.Stream os = req.GetRequestStream())
{
os.Write(bytes, 0, bytes.Length);
}
// read the response
string response;
using (System.Net.WebResponse resp = req.GetResponse())
{
if (resp == null) return;
using (System.IO.StreamReader sr = new System.IO.StreamReader(resp.GetResponseStream()))
{
response = sr.ReadToEnd().Trim();
}
}
// the variable response holds the results of the request...
Credits: Hanselman, Simon (SO Question)

This is the RoR application telling you that you have not formed a request that it can handle; the destination script exists (otherwise you'd see a 404), the request is being handled (otherwise you'd get a 400 error) and it's been encoded correctly (or you'd get a 415 error) but the actual instruction can't be carried out.
Looking at it, you seem to be loading some email information. The RoR application could be telling you that the username and password is wrong, or that the user doesn't exist, or something else. It's up to the RoR application itself.
I think the code itself is good; it's just that the app at the other end isn't happy about doing what you ask it. Are you missing something else in the request information, like a command? (eg command=getnetemails&email=e2#email.com&password=hunter2) Are you sure the email/password combination you are passing is good?
see here for more on the 422 error.

Add the below line above your code.
System.Net.ServicePointManager.Expect100Continue = false;
Are you trying to access an authentication required page?

it was solved by returning xml instead of just unstructured text on the RoR side

Twitter: verifying username and password in C#

Bounty Question
I am using c# 3.5 Window Forms Application. I am using the code mentioned in the accepted answer. and I am getting below error
The remote server returned an error: (401) Unauthorized.
Sample code to verify the UserName and Password will be really appreciated
Bounty Question Ends
I have an application with the following use-case: when the user first starts using the application, he inputs his username and password. Then, at a much later stage, the application may update his status.
Currently I'm using Twitterizer, but I believe the question is beyond the scope of the specific library I'm using. Following are the two relevant lines of code:
Twitter twitter = new Twitter("username", "password", "source");
twitter.Status.Update("update");
The construction of the Twitter object does not throw an exception if the username/password are incorrect. This is probably because nothing is sent at this point. On the other hand, the status update does throw an exception if the username/password are invalid.
My problem is that I want to validate the username/password at the point of user input, not when trying to post the update.
How can I validate the username/password without posting anything (in Twitterizer or otherwise)?

Taking a quick look at the verify_credentials API as mentioned by peSHIr, I wrote a little routine which seems to do the trick. It's late, but I was able to test it a couple of times and seems to work.
In my function, I am just returning true if I I get an HttpResponseCode.OK, and false if I get anything else or an exception is thrown. If twitter does not like the uid/password an exception will be thrown with a 401 error (not authorized.)
public bool CheckTwitterCredentials(string UserName, string Password)
{
// Assume failure
bool Result = false;
// A try except block to handle any exceptions
try {
// Encode the user name with password
string UserPass = Convert.ToBase64String(
System.Text.Encoding.UTF8.GetBytes(UserName + ":" + Password));
// Create our HTTP web request object
HttpWebRequest Request =
(HttpWebRequest)WebRequest.Create("http://twitter.com/account/verify_credentials.xml");
// Set up our request flags and submit type
Request.Method = "GET";
Request.ContentType = "application/x-www-form-urlencoded";
// Add the authorization header with the encoded user name and password
Request.Headers.Add("Authorization", "Basic " + UserPass);
// Use an HttpWebResponse object to handle the response from Twitter
HttpWebResponse WebResponse = (HttpWebResponse)Request.GetResponse();
// Success if we get an OK response
Result = WebResponse.StatusCode == HttpStatusCode.OK;
} catch (Exception Ex) {
System.Diagnostics.Debug.WriteLine("Error: " + Ex.Message);
}
// Return success/failure
return Result;
}

You could try to use the API call account/verify_credentials. Hopefully the API library you use already supports this.. Twitter is notorious now for really hating third party programmers, so unless you have good reason to do something with Twitter, just stay away...

I have used other Twitter Libraries but none of them support checking the username and password for validitity. This might be because Twitter API does not have the facility to validate the username and password unless we try to do something which requires authentication.
One thing you can do is try to get friend list or any other methods that requires authentication.

Twitter API hasn't supported username/password in years. Instead, you have OAuth, which lets the user authorize your application to act on their behalf. Twitter has an account/verify_credentials endpoint you can use to verify whether the user who's tokens you have still authorizes your app. Here's an example of how you could call this endpoint with LINQ to Twitter:
var accounts =
from acct in twitterCtx.Account
where acct.Type == AccountType.VerifyCredentials
select acct;
You can visit Account/VerifyCredentials documentation for more details:

as #joe-mayo informed, you have to switch to OAuth. twitter expired v1 of their API and they documented that in following url https://dev.twitter.com/docs/faq#17750.

Here's a function i wrote that will verify twitter username and password in C# :
public bool isTwitterValid(string username, string password)
{
try
{
string user = Convert.ToBase64String(System.Text.Encoding.UTF8.GetBytes(username + ":" + password));
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("https://twitter.com/statuses/verify.xml");
request.Method = "POST";
request.ServicePoint.Expect100Continue = false;
request.Headers.Add("Authorization", "Basic " + user);
request.ContentType = "application/x-www-form-urlencoded";
WebResponse response = request.GetResponse();
StreamReader reader = new StreamReader(response.GetResponseStream());
string responseString = reader.ReadToEnd();
reader.Close();
}
catch (Exception ex)
{
if (ex.Message.Contains("404")) { return true; }
}
return false;
}