I am trying to write some code to connect to an HTTPS site that uses Siteminder authentication.
I keep getting a 401. Any ideas?
I have read a few different things on here but none have really seemed all that helpful. I am also using Fiddler/Firefox Tamper to snoop what's going on.
Here is what I've got so far in regards to code:
try
{
Uri uri = new Uri("https://websiteaddresshere");
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uri) as HttpWebRequest;
request.Accept = "text/html, application/xhtml+xml, */*";
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko";
// request.Connection = "Keep-Alive";
// request.Method = "Get";
// request.Accept = "text";
request.AllowAutoRedirect = true;
request.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
Cookie emersoncookie = new Cookie("SMCHALLENGE", "YES");
emersoncookie.Domain = "mydomain";
emersoncookie.Path = "/";
// authentication
var cache = new CredentialCache();
cache.Add(uri, "False", new NetworkCredential("myusername", "mypassword"));
request.Credentials = cache;
// response.
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
using (Stream stream = response.GetResponseStream())
{
XmlTextReader reader = new XmlTextReader(stream);
MessageBox.Show(stream.ToString());
}
}
}
catch (WebException exception)
{
string responseText;
using (var reader = new StreamReader(exception.Response.GetResponseStream()))
{
responseText = reader.ReadToEnd();
MessageBox.Show(responseText.ToString());
}
}
After doing some more reading on the MSDN website I decided to go a different route.
I ended up making this a service since it will need to be a service at the end of the day and did the following:
CookieContainer emersoncookie = new CookieContainer();
HttpWebRequest request = (HttpWebRequest)
WebRequest.Create("https://websiteaddress");
request.Credentials = new NetworkCredential("username", "password");
request.CookieContainer = emersoncookie;
request.Method = "GET";
HttpWebResponse response = (HttpWebResponse)
request.GetResponse();
Stream resStream = response.GetResponseStream();
using (Stream output = File.OpenWrite(#"c:\\somefolder\\somefile.someextention"))
using (Stream input = resStream)
{
input.CopyTo(output);
}
To anyone that might be running into Siteminder authentication issues, this piece of code works pretty well.
I couldn't get Jasen's code to work. Maybe your SM setup is different from mine. But with SiteMinder it's generally a two step authentication process. The code block below works for me:
//Make initial request
RestClient client = new RestClient("http://theResourceDomain/myApp");
client.CookieContainer = new CookieContainer();
IRestResponse response = client.Get(new RestRequest("someProduct/orders"));
//Now add credentials.
client.Authenticator = new HttpBasicAuthenticator("username", "password");
//Get resource from the SiteMinder URI which will redirect to the correct API URI upon authentication.
response = client.Get(new RestRequest(response.ResponseUri));
Although this uses RestSharp, it can be easily replicated using HttpClient or even HttpWebRequest.
Related
I have a C# code (it's a Web Application, that's hosted on IIS) where I use HttpWebRequest to get HttpWebResponse. There I make request to any website & get the response as string, then I analysis the response string. But recently I get the response where JavaScript performs data fetching after page is loaded in browser.
I tried to debug this in firebug & saw that at bottom of response there's a JavaScript function that updates the dom elements after pageload. Is there any way that I could do the same in my C# code. I have searched on net about this found not solution till now.
Following is the code I am using:
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
foreach (Cookie cook in response.Cookies)
{
Cookie cookie = new Cookie();
cookie.Name = cook.Name;
cookie.Value = cook.Value;
cookie.Domain = cook.Domain;
cookie.Expires = DateTime.Now.AddDays(10);
cookieList.Add(cookie);
}
string postData = string.Format("username=" + txtUserID.Text + "&password=" + txtPwd.Text + "&url=https://example.com/&game=");
byte[] postBytes = Encoding.UTF8.GetBytes(postData);
HttpWebRequest req = (HttpWebRequest)WebRequest.Create("https://login.example.com/Login/authenticate");
req.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:42.0) Gecko/20100101 Firefox/42.0";
req.KeepAlive = true;
req.AutomaticDecompression = DecompressionMethods.GZip;
////set the cookie
req.CookieContainer = new CookieContainer();
foreach (Cookie cook in cookieList)
{
Cookie cookie = new Cookie();
cookie.Name = cook.Name;
cookie.Value = cook.Value;
cookie.Domain = cook.Domain;
cookie.Expires = DateTime.Now.AddDays(10);
req.CookieContainer.Add(cookie);
}
req.Headers.Add("Accept-Encoding", "gzip, deflate");
req.Headers.Add("Accept-Language", "en-GB,en-US;q=0.8,en;q=0.6");//en-GB,en-US;q=0.8,en;q=0.6
req.Method = "POST";
req.Host = "login.example.com";
req.Referer = "https://login.example.com/Login/logout";
req.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8";
req.ContentType = "application/x-www-form-urlencoded;";
req.ContentLength = postBytes.Length;
//getting the request stream and posting data
StreamWriter requestwriter = new StreamWriter(req.GetRequestStream(), System.Text.Encoding.ASCII);
requestwriter.Write(postData);
requestwriter.Close();
HttpWebResponse myHttpWebResponse = (HttpWebResponse)req.GetResponse();
Stream responseStream = myHttpWebResponse.GetResponseStream();
StreamReader myStreamReader = new StreamReader(responseStream, Encoding.ASCII);
string responseString = myStreamReader.ReadToEnd();
myStreamReader.Close();
responseStream.Close();
myHttpWebResponse.Close();
I finally got the easy solution to my need. following is the link that I followed:
Link to tutorial
Following is the code that'll get the results:
First you'll need to import following:
using System.Drawing;
using OpenQA.Selenium;
using OpenQA.Selenium.PhantomJS;
using System.Text.RegularExpressions;
using System.IO;
using HtmlAgilityPack;
Now the code:
var options = new PhantomJSOptions();
var driver = new PhantomJSDriver(options);
driver.Manage().Window.Size = new Size(1360, 728);
var size = driver.Manage().Window.Size;
driver.Navigate().GoToUrl("https://example.com/");
string url = driver.Url;
//the driver can now provide you with what you need (it will execute the script)
//get the source of the page
var source = driver.PageSource;
//fully navigate the dom
var pathElement1 = driver.FindElementByName("username");
var pathElement2 = driver.FindElementByName("password");
var pathElement3 = driver.FindElementByXPath("//button[#class='SubmitButton']");
pathElement1.Clear();
pathElement1.SendKeys("username");
pathElement2.Clear();
pathElement2.SendKeys("password");
pathElement3.Click();
//Now get the response after login button click
source = driver.PageSource;
I'm trying to login on facebook and retrive a token by using this link:
https://www.facebook.com/dialog/oauth?client_id=282892925078054&redirect_uri=https://www.facebook.com/&response_type=token
My code looks like this, but i get an invalid link when i'm requesting the link above.
CookieCollection cookies = new CookieCollection();
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("https://www.facebook.com/login.php?login_attempt=1");
request.CookieContainer = new CookieContainer();
request.CookieContainer.Add(cookies);
//Get the response from the server and save the cookies from the first request..
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
cookies = response.Cookies;
string getUrl = "https://www.facebook.com/login.php?login_attempt=1";
string postData = String.Format("email={0}&pass={1}", email, pass);
HttpWebRequest getRequest = (HttpWebRequest)WebRequest.Create(getUrl);
getRequest.CookieContainer = new CookieContainer();
getRequest.CookieContainer.Add(cookies); //recover cookies First request
getRequest.Method = WebRequestMethods.Http.Post;
getRequest.UserAgent = "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.121 Safari/535.2";
getRequest.AllowWriteStreamBuffering = true;
getRequest.ProtocolVersion = HttpVersion.Version11;
getRequest.AllowAutoRedirect = true;
getRequest.ContentType = "application/x-www-form-urlencoded";
byte[] byteArray = Encoding.ASCII.GetBytes(postData);
getRequest.ContentLength = byteArray.Length;
Stream newStream = getRequest.GetRequestStream(); //open connection
newStream.Write(byteArray, 0, byteArray.Length); // Send the data.
newStream.Close();
HttpWebResponse getResponse = (HttpWebResponse)getRequest.GetResponse();
cookies = getResponse.Cookies;
using (StreamReader sr = new StreamReader(getResponse.GetResponseStream()))
{
string sourceCode = sr.ReadToEnd();
}
//Get the token
HttpWebRequest webRequest = (HttpWebRequest)WebRequest.Create("https://www.facebook.com/dialog/oauth?client_id=282892925078054&redirect_uri=https://www.facebook.com/&response_type=token");
getRequest.CookieContainer = new CookieContainer();
getRequest.CookieContainer.Add(cookies);
webRequest.AllowAutoRedirect = false;
HttpWebResponse rresponse = (HttpWebResponse)webRequest.GetResponse();
if (rresponse.StatusCode == HttpStatusCode.Redirect)
{
Console.WriteLine("redirected to: " + rresponse.GetResponseHeader("Location"));
}
Please help me. Thanks in advance.
https://www.facebook.com/dialog/oauth?client_id=282892925078054&redirect_uri=https://www.facebook.com/&response_type=token
The URL does not allow the application configuration.: The application settings do not allow one or more of the URLs. Internet Site URL and Canvas URL or domain URLs must be sub-domains of application domains.
Trying to use C# to POST HttpWebRequest into airframes.org for aircraft information. This is the code I use for many other POST request with no problems (used with other urls), but it / I am not able to load the airframes.org page using ICAO24 number (A64294).
var cookies = new CookieContainer();
ServicePointManager.Expect100Continue = false;
var request = (HttpWebRequest)WebRequest.Create("http://www.airframes.org/");
request.CookieContainer = cookies;
request.Method = "POST";
request.KeepAlive = true;
request.ContentType = "application/x-www-form-urlencoded";
using (var requestStream = request.GetRequestStream())
using (var writer = new StreamWriter(requestStream))
{
writer.Write("reg=&selcal=&icao24=A64294&submit=submit");
}
using (var responseStream = request.GetResponse().GetResponseStream())
using (var reader = new StreamReader(responseStream))
{
var result = reader.ReadToEnd();
Console.WriteLine(result);
}
Note that the site has a no-bots policy, which is why your request won't work.
That being said, if you still wish to request the page, adding a user-agent string (so your request looks like it came from a browser) does the trick:
request.UserAgent = "Mozilla/6.0 (Windows NT 6.2; WOW64; rv:16.0.1) Gecko/20121011 Firefox/16.0.1";
It is usually a good idea to respect the policies of a site. The code above is merely for educational purposes.
I have some code that logs me into Facebook and it works fine as far as returning me the page you see right after you log in.
But I'm trying to request my actual profile after I log in rather than just my news feed.
So I'd have to send my email and password to the login page, then request my profile.
How can I keep the login data around while requesting my profile?
Here is what I've got
public static string logIn()
{
//get the cookies before you try to log in
CookieCollection cookies = new CookieCollection();
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("https://www.facebook.com");
request.CookieContainer = new CookieContainer();
request.CookieContainer.Add(cookies);
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
cookies = response.Cookies;
response.Close();
//logging in
HttpWebRequest getRequest = (HttpWebRequest)WebRequest.Create("https://www.facebook.com/login.php?login_attempt=1");
getRequest.CookieContainer = new CookieContainer();
getRequest.CookieContainer.Add(cookies);
getRequest.Method = WebRequestMethods.Http.Post;
getRequest.UserAgent = "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.121 Safari/535.2";
getRequest.AllowWriteStreamBuffering = true;
getRequest.ProtocolVersion = HttpVersion.Version11;
getRequest.AllowAutoRedirect = true;
getRequest.ContentType = "application/x-www-form-urlencoded";
//sending the email/password
byte[] byteArray = Encoding.ASCII.GetBytes("email=myemail#yahoo.com&pass=mypassword");
getRequest.ContentLength = byteArray.Length;
Stream newStream = getRequest.GetRequestStream();
newStream.Write(byteArray, 0, byteArray.Length);
newStream.Close();
//returns the source of the page after logging in
HttpWebResponse getResponse = (HttpWebResponse)getRequest.GetResponse();
StreamReader sr = new StreamReader(getResponse.GetResponseStream());
string source = sr.ReadToEnd();
cookies.Add(getResponse.Cookies);
//tries to get my profile source
//everything works fine until here
getRequest = (HttpWebRequest)WebRequest.Create("http://www.facebook.com/myprofile");
getRequest.CookieContainer = new CookieContainer();
getRequest.CookieContainer.Add(cookies);
getRequest.Method = WebRequestMethods.Http.Get;
getRequest.UserAgent = "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.121 Safari/535.2";
getRequest.AllowWriteStreamBuffering = true;
getRequest.ProtocolVersion = HttpVersion.Version11;
getRequest.AllowAutoRedirect = true;
getRequest.ContentType = "application/x-www-form-urlencoded";
getResponse = (HttpWebResponse)getRequest.GetResponse();
sr = new StreamReader(getResponse.GetResponseStream());
source = sr.ReadToEnd();
getResponse.Close();
return source;
}
I've tried several ways of doing this and I have gotten it to return my profile, but it returns it as if I was not logged in and you can't actually view my profile (because it is set to private)
So I need my login info to be included somehow when requesting my profile.
This had me stumped for a while when I was making something similar. You need to make sure you're setting the proper cookies for your request after you've logged in. Try adding the new cookies to the CookieCollection object before applying them to your last request.
// ^ previous code ^
HttpWebResponse getResponse = (HttpWebResponse)getRequest.GetResponse();
StreamReader sr = new StreamReader(getResponse.GetResponseStream());
string source = sr.ReadToEnd();
Uri facebookPage = new Uri("http://www.facebook.com"); // .GetCookies() only accepts a Uri
cookies.Add(request.CookieContainer.GetCookies(facebookPage));
getResponse.Close(); // Always make sure you close your responses
// V requesting your profile code V
You have to start a CookieCollection at the beginning and use it for all requests.
A CookieCollection updates itself with new cookies whenever you do a request so using cookies.Add() was also messing it up.
But I got it working fine now.
I am trying to authenticate google with the following code but google sent me back to the login page again.
//STEP# 1
string loginURL = "https://www.google.com/accounts/ServiceLoginBox?service=analytics&nui=1&hl=en-US&continue=https%3A%2F%2Fwww.google.com%2Fanalytics%2Fsettings%2F%3Fet%3Dreset%26hl%3Den%26et%3Dreset%26hl%3Den-US";
request = (HttpWebRequest)WebRequest.Create(loginURL);
request.CookieContainer = cookieJar;
request.Method = "GET";
request.KeepAlive = true;
request.UserAgent = "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.4) Gecko/2008111217 Fedora/3.0.4-1.fc10 Firefox/3.0.4";
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
foreach (Cookie cook in response.Cookies)
{
cookieJar.Add(cook);
}
using (StreamReader sr = new StreamReader(response.GetResponseStream()) )
{
serverResponse = sr.ReadToEnd();
sr.Close();
}
galx = ExtractValue(serverResponse,"GALX","name=\"GALX\" value=\"");
Console.WriteLine(galx);
//Request# 2
string uriWithData = "https://www.google.com/accounts/ServiceLoginBoxAuth";
request = (HttpWebRequest)WebRequest.Create(uriWithData);
request.KeepAlive = true;
request.UserAgent = "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.4) Gecko/2008111217 Fedora/3.0.4-1.fc10 Firefox/3.0.4";
request.Method = "POST";
request.CookieContainer = cookieJar;
string param = string.Format("Email={0}&Passwd={1}&continue={2}&service=analytics&nui=1&dsh=8209101995200094904&GALX={3}&hl=en-US&PersistentCookie=yes","**my email address**",p,"",galx);
byte[] postArr = StrToByteArray(param);
request.ContentType = #"application/x-www-form-urlencoded";
request.ContentLength = param.Length;
Stream reqStream = request.GetRequestStream();
reqStream.Write(postArr,0,postArr.Length);
reqStream.Close();
response = (HttpWebResponse)request.GetResponse();
foreach (Cookie cook in response.Cookies)
{
cookieJar.Add(cook);
}
using (StreamReader sr = new StreamReader(response.GetResponseStream()) )
{
serverResponse = sr.ReadToEnd();
Console.WriteLine(serverResponse);
// Close and clean up the StreamReader
sr.Close();
}
I have no idea an I am pretty sure not many people are going to want to sift through that much code.
One thing that I see at a glance that may be causing problems is that you are playing with the cookies too much.
Create one CookieContainer and just pass it in with each request.
No need to 'transfer' or 'recreate' the container.
Try looking into google accounts api and GBaseService. I believe you will have to set HOSTED_OR_GOOGLE in accountType as Marty has done here in this post.