We Keep Coding

login to ajax web page from c# code

i'm trying to log in a site with username + password through a c# code.
i found out that it uses Ajax to authenticate...
how should i implement such login ?
the elements in the web page doesn't seem to have an "id"...
i tried to implement it using HtmlAgilityPack but i don't think this is the correct direction...
i can't simulate a click button since i don't find "id" for the button.
if (tableNode.Attributes["class"].Value == "loginTable")
{
var userInputNode =
tableNode.SelectSingleNode("//input[#data-logon-popup-form-user-name-input='true']");
var passwordInputNode =
tableNode.SelectSingleNode("//input[#data-logon-popup-form-password-input='true']");
userInputNode.SetAttributeValue("value", "myemail#gmail.com");
passwordInputNode.SetAttributeValue("value", "mypassword");
var loginButton = tableNode.SelectSingleNode("//div[#data-logon-popup-form-submit-btn='true']");
}

This question is quite broad but I'll help you in the general direction:
Use Chrome DevTools (F12) => Network tab => Check the "Preserve Log". An alternative could be Fiddler2
Login manually and look at the request the AJAX sends. Save the endpoint (the URL) and save the Body of the request (the Json data that's in the request with username and password)
Do the post directly in your C# code and forget about HtmlAgilityPack unless you need to actually get some dynamic data from the page, but that's rarely the case
Login with something like this code snippet: POSTing JSON to URL via WebClient in C#
Now you're logged in. You usually receive some data from the server when you're logging in, so save it and use it for whatever you want to do next. I'm guessing it might have some SessionId or some authentication token that your future requests will need as a parameter to prove that you're actually logged in.

HttpWebRequest: Add Cookie to CookieContainer -> ArgumentException (Parametername: cookie.Domain)

I'm trying to login to a website via my application.
What I did:
First I figured out how the browser does the authorization process with Fiddler.
I examined how the POST request is built and I tried to reconstruct it.
The browser sends 4 cookies (Google Analytics) and I tried to set them:
CookieContainer gaCookies = new CookieContainer();
gaCookies.Add(new Cookie("__utma", "#########.###########.##########.##########.##########.#"));
gaCookies.Add(new Cookie("__utmb", "#########.#.##.##########"));
gaCookies.Add(new Cookie("__utmc", "#########"));
gaCookies.Add(new Cookie("__utmz", "#########.##########.#.#.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)"));
(just replaced the original cookie data with #)
Then I went through the code with the debugger and as soon as the first gaCookies.Add is executed, the application stops with an
System.ArgumentException: The parameter '{0}' cannot be an empty string. Parameter name: cookie.Domain
I would like to know why this happens. The constructor of Cookie doesn't require a domain and I don't know where I can get this value?
Would be very great if someone of you could help me with this.
I'm not a webdeveloper or an expert in web stuff so I don't know much about it.
Is there maybe a great source where I can learn about this if there is no "short and quick answer"?

CookieContainers can hold multiple cookies for different websites, therefor a label (the Domain) has to be provided to bind each cookie to each website. The Domain can be set when instantiating the individual cookies like so:
Cookie chocolateChip = new Cookie("CookieName", "CookieValue") { Domain = "DomainName" };
An easy way to grab the domain to is to make a Uri (if you aren't using one already) that contains your target url and set the cookie's domain using the Uri.Host property.
CookieContainer gaCookies = new CookieContainer();
Uri target = new Uri("http://www.google.com/");
gaCookies.Add(new Cookie("__utmc", "#########") { Domain = target.Host });

How to pass query string parameter in asp.net?

I am using Access Control service (ACS). I fetched all identity providers (ip) which i set for my application using the following code :
public ActionResult IdentityProviders(string serviceNamespace, string appId)
{
string idpsJsonEndpoint = string.Format(Global.IdentityProviderJsonEndpoint, serviceNamespace, appId);
var client = new WebClient();
var data = client.DownloadData(idpsJsonEndpoint);
return Content(Encoding.UTF8.GetString(data), "application/json");
}
When user click over the signin link the above code called using ajax and get the ips and display them in jquery-ui dialog. And when user click any one of the ips for login the browser redirect to the selected ip login page. After successful login the control return to my control which i set as a returnUrl. Upto this every thing is works fine.
Now what i am trying to do is to pass some values to identity provider (ip) login page and want to get back those values at my returnUrl controller. For this i searched and came to know that there is a query string parameter known as wctx which we can set and get the value at return url. But i dont know how to do this. Can anybody please guid me how can i achieve this?

It is relatively (pretty) easy.
Your URL for listing IdPs looks something like this:
https://[your_namespace].accesscontrol.windows.net:443/v2/metadata/IdentityProviders.js?protocol=wsfederation&realm=[your_realm]&reply_to=[configured_return_url_for_your_rp]&context=&request_id=&version=1.0&callback=
This is the most complete request for list of Identity Providers. Your may miss some variables (such as context, or reply_to), but what I show is the complete request.
So now you have two options:
inclide your own reply_to parameter. It must be withing the configured realm. So if your realm is https://www.mygreatapp.com/, your default return URL would probably be something like https://www.mygreatapp.com/returnUrl/ (if your controller to handle ACS response is returnUrlController. Now, you can safely change the reply_to to be https://www.mygreatapp.com/returnUrl/?foo=bar, just make sure you URL Encode the query string.
Use the context parameter. It is safer to use and I would suggest using it. Now your URL for fetching list of IdPs will be something like:
https://[your_namespace].accesscontrol.windows.net:443/v2/metadata/IdentityProviders.js?protocol=wsfederation&realm=[your_realm]&reply_to=[configured_return_url_for_your_rp]&context=[your_custom_string_value_which_you_may_even_encrypt]&request_id=&version=1.0&callback=
Note the now there is context value present in the request for IdP list ([your_custom_string_value_which_you_may_even_encrypt]). In your returnUrl handler controller, you can check for it with code similar (or equal) to the following:
if (ControllerContext.HttpContext.Request.Form["wresult"] != null)
{
// This is a response from the ACS - you can further inspect the message if you will
SignInResponseMessage message =
WSFederationMessage.CreateFromNameValueCollection(
WSFederationMessage.GetBaseUrl(ControllerContext.HttpContext.Request.Url),
ControllerContext.HttpContext.Request.Form)
as SignInResponseMessage;
if (!string.IsNullOrWhiteSpace(message.Context))
{
// do whatever you want with the context value
}
}
You may want to perform any/more additional checks while handling the SignInResponse from ACS.

OAuth authentication without browser [duplicate]

I'm trying to create a .NET-based client app (in WPF - although for the time being I'm just doing it as a console app) to integrate with an OAuth-enabled application, specifically Mendeley (http://dev.mendeley.com), which apparently uses 3-legged OAuth.
This is my first time using OAuth, and I'm having a lot of difficulty getting started with it. I've found several .NET OAuth libraries or helpers, but they seem to be more complicated than I think I need. All I want to do is be able to issue REST requests to the Mendeley API and get responses back!
So far, I've tried:
DotNetOpenAuth
http://github.com/bittercoder/DevDefined.OAuth
http://oauth.googlecode.com/svn/code/csharp/
The first (DotNetOpenAuth) seems like it could possibly do what I needed if I spent hours and hours trying to work out how. The second and third, as best I can tell, don't support the verification codes that Mendeley is sending back -- although I could be wrong about this :)
I've got a consumer key and secret from Mendeley, and with DotNetOpenAuth I managed to get a browser launched with the Mendeley page providing a verification code for the user to enter into the application. However, at this point I got lost and couldn't work out how to sensibly provide that back to the application.
I'm very willing to admit that I have no idea where to start with this (although it seems like there's quite a steep learning curve) - if anyone can point me in the right direction I'd appreciate it!

I agree with you. The open-source OAuth support classes available for .NET apps are hard to understand, overly complicated (how many methods are exposed by DotNetOpenAuth?), poorly designed (look at the methods with 10 string parameters in the OAuthBase.cs module from that google link you provided - there's no state management at all), or otherwise unsatisfactory.
It doesn't need to be this complicated.
I'm not an expert on OAuth, but I have produced an OAuth client-side manager class, that I use successfully with Twitter and TwitPic. It's relatively simple to use. It's open source and available here: Oauth.cs
For review, in OAuth 1.0a...kinda funny, there's a special name and it looks like a "standard" but as far as I know the only service that implements "OAuth 1.0a" is Twitter. I guess that's standard enough. ok, anyway in OAuth 1.0a, the way it works for desktop apps is this:
You, the developer of the app, register the app and get a "consumer key" and "consumer secret". On Arstechnica, there's a well written analysis of why this model isn't the best, but as they say, it is what it is.
Your app runs. The first time it runs, it needs to get the user to explicitly grant approval for the app to make oauth-authenticated REST requests to Twitter and its sister services (like TwitPic). To do this you must go through an approval process, involving explicit approval by the user. This happens only the first time the app runs. Like this:
request a "request token". Aka temporary token.
pop a web page, passing that request token as a query param. This web page presents UI to the user, asking "do you want to grant access to this app?"
the user logs in to the twitter web page, and grants or denies access.
the response html page appears. If the user has granted access, there's a PIN displayed in a 48-pt font
the user now needs to cut/paste that pin into a windows form box, and click "Next" or something similar.
the desktop app then does an oauth-authenticated request for an "Access token". Another REST request.
the desktop app receives the "access token" and "access secret".
After the approval dance, the desktop app can just use the user-specific "access token" and "access secret" (along with the app-specific "consumer key" and "consumer secret") to do authenticated requests on behalf of the user to Twitter. These don't expire, although if the user de-authorizes the app, or if Twitter for some reason de-authorizes your app, or if you lose your access token and/or secret, you'd need to do the approval dance again.
If you're not clever, the UI flow can sort of mirror the multi-step OAuth message flow. There is a better way.
Use a WebBrowser control, and open the authorize web page within the desktop app. When the user clicks "Allow", grab the response text from that WebBrowser control, extract the PIN automatically, then get the access tokens. You send 5 or 6 HTTP requests but the user needs to see only a single Allow/Deny dialog. Simple.
Like this:
If you've got the UI sorted, the only challenge that remains is to produce oauth-signed requests. This trips up lots of people because the oauth signing requirements are sort of particular. That's what the simplified OAuth Manager class does.
Example code to request a token:
var oauth = new OAuth.Manager();
// the URL to obtain a temporary "request token"
var rtUrl = "https://api.twitter.com/oauth/request_token";
oauth["consumer_key"] = MY_APP_SPECIFIC_KEY;
oauth["consumer_secret"] = MY_APP_SPECIFIC_SECRET;
oauth.AcquireRequestToken(rtUrl, "POST");
THAT'S IT. Simple. As you can see from the code, the way to get to oauth parameters is via a string-based indexer, something like a dictionary. The AcquireRequestToken method sends an oauth-signed request to the URL of the service that grants request tokens, aka temporary tokens. For Twitter, this URL is "https://api.twitter.com/oauth/request_token". The oauth spec says you need to pack up the set of oauth parameters (token, token_secret, nonce, timestamp, consumer_key, version, and callback), in a certain way (url-encoded and joined by ampersands), and in a lexicographically-sorted order, generate a signature on that result, then pack up those same parameters along with the signature, stored in the new oauth_signature parameter, in a different way (joined by commas). The OAuth manager class does this for you automatically. It generates nonces and timestamps and versions and signatures automatically - your app doesn't need to care or be aware of that stuff. Just set the oauth parameter values and make a simple method call. the manager class sends out the request and parses the response for you.
Ok, then what? Once you get the request token, you pop the web browser UI in which the user will explicitly grant approval. If you do it right, you'll pop this in an embedded browser. For Twitter, the URL for this is "https://api.twitter.com/oauth/authorize?oauth_token=" with the oauth_token appended. Do this in code like so:
var url = SERVICE_SPECIFIC_AUTHORIZE_URL_STUB + oauth["token"];
webBrowser1.Url = new Uri(url);
(If you were doing this in an external browser you'd use System.Diagnostics.Process.Start(url).)
Setting the Url property causes the WebBrowser control to navigate to that page automatically.
When the user clicks the "Allow" button a new page will be loaded. It's an HTML form and it works the same as in a full browser. In your code, register a handler for the DocumentedCompleted event of the WebBrowser control, and in that handler, grab the pin:
var divMarker = "<div id=\"oauth_pin\">"; // the div for twitter's oauth pin
var index = webBrowser1.DocumentText.LastIndexOf(divMarker) + divMarker.Length;
var snip = web1.DocumentText.Substring(index);
var pin = RE.Regex.Replace(snip,"(?s)[^0-9]*([0-9]+).*", "$1").Trim();
That's a bit of HTML screen scraping.
After grabbing the pin, you don't need the web browser any more, so:
webBrowser1.Visible = false; // all done with the web UI
...and you might want to call Dispose() on it as well.
The next step is getting the access token, by sending another HTTP message along with that pin. This is another signed oauth call, constructed with the oauth ordering and formatting I described above. But once again this is really simple with the OAuth.Manager class:
oauth.AcquireAccessToken(URL_ACCESS_TOKEN,
"POST",
pin);
For Twitter, that URL is "https://api.twitter.com/oauth/access_token".
Now you have access tokens, and you can use them in signed HTTP requests. Like this:
var authzHeader = oauth.GenerateAuthzHeader(url, "POST");
...where url is the resource endpoint. To update the user's status, it would be "http://api.twitter.com/1/statuses/update.xml?status=Hello".
Then set that string into the HTTP Header named Authorization.
To interact with third-party services, like TwitPic, you need to construct a slightly different OAuth header, like this:
var authzHeader = oauth.GenerateCredsHeader(URL_VERIFY_CREDS,
"GET",
AUTHENTICATION_REALM);
For Twitter, the values for the verify creds url and realm are "https://api.twitter.com/1/account/verify_credentials.json", and "http://api.twitter.com/" respectively.
...and put that authorization string in an HTTP header called X-Verify-Credentials-Authorization. Then send that to your service, like TwitPic, along with whatever request you're sending.
That's it.
All together, the code to update twitter status might be something like this:
// the URL to obtain a temporary "request token"
var rtUrl = "https://api.twitter.com/oauth/request_token";
var oauth = new OAuth.Manager();
// The consumer_{key,secret} are obtained via registration
oauth["consumer_key"] = "~~~CONSUMER_KEY~~~~";
oauth["consumer_secret"] = "~~~CONSUMER_SECRET~~~";
oauth.AcquireRequestToken(rtUrl, "POST");
var authzUrl = "https://api.twitter.com/oauth/authorize?oauth_token=" + oauth["token"];
// here, should use a WebBrowser control.
System.Diagnostics.Process.Start(authzUrl); // example only!
// instruct the user to type in the PIN from that browser window
var pin = "...";
var atUrl = "https://api.twitter.com/oauth/access_token";
oauth.AcquireAccessToken(atUrl, "POST", pin);
// now, update twitter status using that access token
var appUrl = "http://api.twitter.com/1/statuses/update.xml?status=Hello";
var authzHeader = oauth.GenerateAuthzHeader(appUrl, "POST");
var request = (HttpWebRequest)WebRequest.Create(appUrl);
request.Method = "POST";
request.PreAuthenticate = true;
request.AllowWriteStreamBuffering = true;
request.Headers.Add("Authorization", authzHeader);
using (var response = (HttpWebResponse)request.GetResponse())
{
if (response.StatusCode != HttpStatusCode.OK)
MessageBox.Show("There's been a problem trying to tweet:" +
Environment.NewLine +
response.StatusDescription);
}
OAuth 1.0a is sort of complicated under the covers, but using it doesn't need to be.
The OAuth.Manager handles the generation of outgoing oauth requests, and the receiving and processing of oauth content in the responses. When the Request_token request gives you an oauth_token, your app doesn't need to store it. The Oauth.Manager is smart enough to do that automatically. Likewise when the access_token request gets back an access token and secret, you don't need to explicitly store those. The OAuth.Manager handles that state for you.
In subsequent runs, when you already have the access token and secret, you can instantiate the OAuth.Manager like this:
var oauth = new OAuth.Manager();
oauth["consumer_key"] = CONSUMER_KEY;
oauth["consumer_secret"] = CONSUMER_SECRET;
oauth["token"] = your_stored_access_token;
oauth["token_secret"] = your_stored_access_secret;
...and then generate authorization headers as above.
// now, update twitter status using that access token
var appUrl = "http://api.twitter.com/1/statuses/update.xml?status=Hello";
var authzHeader = oauth.GenerateAuthzHeader(appUrl, "POST");
var request = (HttpWebRequest)WebRequest.Create(appUrl);
request.Method = "POST";
request.PreAuthenticate = true;
request.AllowWriteStreamBuffering = true;
request.Headers.Add("Authorization", authzHeader);
using (var response = (HttpWebResponse)request.GetResponse())
{
if (response.StatusCode != HttpStatusCode.OK)
MessageBox.Show("There's been a problem trying to tweet:" +
Environment.NewLine +
response.StatusDescription);
}
You can download a DLL containing the OAuth.Manager class here. There is also a helpfile in that download. Or you can view the helpfile online.
See an example of a Windows Form that uses this manager here.
WORKING EXAMPLE
Download a working example of a command-line tool that uses the class and technique described here:

Grabbing Cookies in Web Browser Control - WP7

In order to log into a certain part of a website the users of my application require their cookie. To do this I need to grab it and pass it to url.
Does anyone know how to grab a certain websites cookie from the browser control?
I saw this method but wasn't quite clear.
Thanks, TP.

As of WP 7.1 Mango "release", if one may call it, please see the WebBrowser Control Overview for Windows Phone. It has been recently updated a little bit, and it turns out that they actually have added some support for cookie-retrieval from the WebBrowser. On the bottom of the page you will find a tiny link GetCookies(WebBrowser) pointing to description of a new class: WebBrowserExtensions with this very handy method. Yes, this class has only that one single member. It's an extension method, I suppose no explanations needed on that.
I have not played with this method much, but it seems that this will allow you to access the very same thing as the JS trick: the cookieset for the current URL. It probably will not allow to set anything, nor to peek cookies for other URLs. Maybe if you play hard with the CookieContainer you will receive, but I doubt.
On the 7.0 release, I've been struggling quite hard to achieve "cookie transparency" for my App. Long story short, my app was doing some background HTTP requests, and also had a WebBrowser to show some online content -- and "it would be great" if both sources of connections would emit the same cookies to the server.. And guess what, my application had to make the first request, then let the browser navigate. With such requirements, there was virtually is no way to achieve consistency of the cookies - bah, even with the current new and glorious GetCookie method, I suppose it would be damn hard. So, to the point - it was possible, but needed to use some hidden API, that is present publicitly on the Phone, but is hidden in the SDK. The API is available the (public) class System.Net.Browser.WebRequestCreator, freely available. The quirk is: in the SDK this class has a single public static property "IWebRequestCreate ClientHttp" with a method "Create" that you can use to "factory" your "raw http" connections - in case you dont want to use the WebClient for some reason. On the phone, and on the emulator, there is a second public static property called "IWebRequestCreate BrowserHttp", easily returned by Reflection:
PropertyInfo brwhttp = typeof(System.Net.Browser.WebRequestCreator)
.GetProperty("BrowserHttp")
with this property, you will be able to obtain a "special" internal instance of IWebRequestCreate that is used internally by the WebBrowser. By opening your background HTTP requests with this class, you will get your cookies automatically set as if they were created/sent by the WebBrowser control, but in turn - you will NOT be able to modify http headers, userprovide http user authentication and neither do a few lowlevel things - because all that settings will be synced with the WebBrowser's data stored for current 'system user instance', if I'm allowed to call it as such on the single-user Phone device heh. The interoperation between connections and the WebBrowser works both ways - if your HTTP connection (created with use of the 'hidden property') receives any settings/cookies/etc -- then the WebBrowser will instantly notice them and update its own cache. No cookie/session loss on neither of the sides!
If you need to passively get cookies for your subsequent connections after some first WebBrowser navigation - please use the GetCookie or the JS way.
But if you need your code to be first, and then pass authz to the WebBrowser -- you will probably have to dig deeper and use the above.. It's been hidden, so please resort to the other means first!
..and don't ask me how did I found it or how much time it took :P
have a nice fun with it
//edit: I've just found out, that the BrowserHttp property is a normal Silverlight's way to access the Browser's connection factory, please see BrowserHttp. It seems that it is only has been hidden in the 'miniSilverlight' written for the WP7 platform!

The approach being described in the post you linked is to use the WebBrowser control's InvokeScript method to run some javascript. However the post appears to use a "cookies" collection which doesn't actually exist.
string cookie = myWebBrowser.InvokeScript("document.cookie") as string;
Now for the hard part the string you get contains all pertinent cookie name/value pairs for the page with the values still being Url encoded. You will need to parse the returned string for the value you need.
See document.cookie property documentation.
Edit:
Looking at it fresh instead of relying on the post, InvokeScript invokes named function on the window of the host browser. Hence the page being displayed in the WebBrowser would itself need to include a function like:-
function getCookie() { return document.cookie; }
Then the InvokeScript would look like:-
string cookie = myWebBrowser.InvokeScript("getCookie");

As #quetzalcoatl already suggested, you can use internal instance of WebRequestCreator to share cookies between browser instances and instances of WebRequest. You don't get to access the cookies directly though, I think that's just a security measure by Microsoft.
This code below creates a WebReqeust object, connected to CookieContainer of WebBrowser instance. It then posts to a url to log in the user and store cookies in the container.
After it's done, all browser instances within the app instance will have required set of cookies.
var browser = new WebBrowser();
var brwhttp = typeof (WebRequestCreator).GetProperty("BrowserHttp");
var requestFactory = brwhttp.GetValue(browser, null) as IWebRequestCreate;
var uri = new Uri("https://www.login.com/login-handler");
var req = requestFactory.Create(uri);
req.Method = "POST";
var postParams = new Dictionary<string, string> {
{"username", "turtlepower"},
{"password": "ZoMgPaSSw0Rd1"}
};
req.BeginGetRequestStream(aReq => {
var webRequest = (HttpWebRequest)aReq.AsyncState;
using (var postStream = webRequest.EndGetRequestStream(aReq)) {
// Build your POST request here
var postDataBuilder = new StringBuilder();
foreach (var pair in paramsDict) {
if (postDataBuilder.Length != 0) {
postDataBuilder.Append("&");
}
postDataBuilder.AppendFormat("{0}={1}", pair.Key, HttpUtility.UrlEncode(pair.Value));
}
var bytes = Encoding.UTF8.GetBytes(postDataBuilder.ToString());
postStream.Write(bytes, 0, bytes.Length);
}
// Receive response
webRequest.BeginGetResponse(aResp => {
var webRequest2 = (HttpWebRequest) aResp.AsyncState;
webRequest = (HttpWebRequest)aResp.AsyncState;
string resp;
using (var response = (HttpWebResponse)webRequest2.EndGetResponse(aResp)) {
using (var streamResponse = response.GetResponseStream()) {
using (var streamReader = new System.IO.StreamReader(streamResponse)) {
resp = streamReader.ReadToEnd();
}
}
}
}, webRequest);
}, req);
One of the issues I couldn't solve though was exceptions thrown when server returns 302 - it seems to throw WebException error with "Not found" description.

// Ensure this is set to true BEFORE navigating to the page
webBrowser1.IsScriptEnabled = true;
// Once the page has loaded, you can read the cookie string
string cookieString = webBrowser1.InvokeScript("eval", new string[] { "document.cookie;" }) as string;
The cookieString variable will contain the full cookie for the document. You can then parse the string.

There is an WebBrowser Extension class which is exactly developed for this:
CookieCollection tempCookies = Microsoft.Phone.Controls.WebBrowserExtensions.GetCookies(this.BrowserControl);