I'm trying to login to www.autoscout24.de and retrieve adds and messages. Login form has a random generated hidden input/token. Being new to C#, I've read different tuts about using C# to login to websites and all I found was simple codes that work only in simple login forms (user:pass). I've imagined a 2-step approach: first make a GET request to retrieve needed data and a POST request with login credentials and other needed imputes. Using HtmlAgilityPack I'm passed first step but the second request just returns the login page again instead of "My account" page.
My code:
using System;
using System.IO;
using System.Net;
using System.Text;
namespace WebRequest__custom
{
class Program
{
static void Main(string[] args)
{
CookieContainer _cookies;
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("https://angebot.autoscout24.de/login?fromNavi=myAS24");
WebResponse _response = request.GetResponse();
Stream stream = _response.GetResponseStream();
StreamReader strReader = new StreamReader(stream);
string _cookiesHeader = _response.Headers["Set-cookie"];
_cookies = request.CookieContainer;
string _content = strReader.ReadToEnd();
//Console.WriteLine(_content.Substring(0,500));
var _dom = new HtmlAgilityPack.HtmlDocument();
_dom.LoadHtml(_content);
// Get POST link
var _postLinkNode = _dom.DocumentNode.SelectSingleNode("//*[#id='loginForm']/div[3]/form");
var postLink = _postLinkNode.Attributes["action"].Value;
//Console.WriteLine(postLink);
//get Token
var _tokenNode = _dom.DocumentNode.SelectSingleNode("//*[#id='loginForm']/div[3]/form/input");
var token = _tokenNode.Attributes["value"].Value;
//Console.WriteLine(token);
// Start login request
HttpWebRequest requestLogin = (HttpWebRequest)WebRequest.Create("https://accounts.autoscout24.com"+ postLink);
requestLogin.ContentType = "application/x-www-form-urlencoded";
requestLogin.Method = "POST";
requestLogin.KeepAlive = true;
requestLogin.AllowAutoRedirect = true;
string postData = "&__RequestVerificationToken=" + token;
postData += "&Username=web-cppxt#mail-tester.com";
postData += "&Password=Qwert123!";
postData += "&RememberMeCheckBox=on&RememberMe=true";
byte[] _bytes = Encoding.UTF8.GetBytes(postData);
requestLogin.ContentLength = _bytes.Length;
requestLogin.CookieContainer = _cookies;
using(Stream sr = requestLogin.GetRequestStream())
{
sr.Write(_bytes, 0, _bytes.Length);
}
WebResponse loginResponse = requestLogin.GetResponse();
StreamReader loginStreamReader = new StreamReader(loginResponse.GetResponseStream());
string secondPage = loginStreamReader.ReadToEnd();
Console.WriteLine(secondPage.Substring(0,500));
Console.ReadKey();
}
}
}
Related
I am making a call to a page on my site using webclient. I'm trying to get the result of the webpage put into a pdf so I am trying to get a string representation of the rendered page. The problem is that the request is not authenticated so all I get is a login screen. I have sent the UseDefaultCredentials property to true but I still get the same result. Below is a portion of my code:
WebClient webClient = new WebClient();
webClient.Encoding = Encoding.UTF8;
webClient.UseDefaultCredentials = true;
return Encoding.UTF8.GetString(webClient.UploadValues(link, "POST",form));
You need to give the WebClient object the credentials. Something like this...
WebClient client = new WebClient();
client.Credentials = new NetworkCredential("username", "password");
What kind of authentication are you using? If it's Forms authentication, then at best, you'll have to find the .ASPXAUTH cookie and pass it in the WebClient request.
At worst, it won't work.
Public Function getWeb(ByRef sURL As String) As String
Dim myWebClient As New System.Net.WebClient()
Try
Dim myCredentialCache As New System.Net.CredentialCache()
Dim myURI As New Uri(sURL)
myCredentialCache.Add(myURI, "ntlm", System.Net.CredentialCache.DefaultNetworkCredentials)
myWebClient.Encoding = System.Text.Encoding.UTF8
myWebClient.Credentials = myCredentialCache
Return myWebClient.DownloadString(myURI)
Catch ex As Exception
Return "Exception " & ex.ToString()
End Try
End Function
This helped me to call API that was using cookie authentication. I have passed authorization in header like this:
request.Headers.Set("Authorization", Utility.Helper.ReadCookie("AuthCookie"));
complete code:
// utility method to read the cookie value:
public static string ReadCookie(string cookieName)
{
var cookies = HttpContext.Current.Request.Cookies;
var cookie = cookies.Get(cookieName);
if (cookie != null)
return cookie.Value;
return null;
}
// using statements where you are creating your webclient
using System.Web.Script.Serialization;
using System.Net;
using System.IO;
// WebClient:
var requestUrl = "<API_url>";
var postRequest = new ClassRoom { name = "kushal seth" };
using (var webClient = new WebClient()) {
JavaScriptSerializer serializer = new JavaScriptSerializer();
byte[] requestData = Encoding.ASCII.GetBytes(serializer.Serialize(postRequest));
HttpWebRequest request = WebRequest.Create(requestUrl) as HttpWebRequest;
request.Method = "POST";
request.ContentType = "application/json";
request.ContentLength = requestData.Length;
request.ContentType = "application/json";
request.Expect = "application/json";
request.Headers.Set("Authorization", Utility.Helper.ReadCookie("AuthCookie"));
request.GetRequestStream().Write(requestData, 0, requestData.Length);
using (var response = (HttpWebResponse)request.GetResponse()) {
var reader = new StreamReader(response.GetResponseStream());
var objText = reader.ReadToEnd(); // objText will have the value
}
}
I'm trying to connect Django API with c # but when I connect I face a problem in C#.
Django here I use as a server API and C# as a client.
Errors in C# are "CSRF is missing or incorrect".
using System;
using System.Net;
using System.IO;
using System.Text;
using System.Collections.Generic;
using System.Net.Http;
using System.Collections.Specialized;
using System.Text.RegularExpressions;
namespace ConsoleApplication1
{
class Program
{
private static string json;
public static string url { get; private set; }
static void Main(string[] args)
{
/*
try
{
CookieContainer container = new CookieContainer();
HttpWebRequest request1 = (HttpWebRequest)WebRequest.Create("http://127.0.0.1/api/");
request1.Proxy = null;
request1.CookieContainer = container;
using (HttpWebResponse response1 = (HttpWebResponse)request1.GetResponse())
{
foreach (Cookie cookie1 in response1.Cookies)
{
//Console.WriteLine(response.IsSuccessStatusCode);
var csrf = cookie1.Value;
Console.WriteLine(csrf);
Console.WriteLine("name=" + cookie1.Name);
Console.WriteLine();
Console.Write((int)response1.StatusCode);
//PostRespone("http://localhost/api/multiplyfunction/");
}
}
}
catch (WebException e)
{
Console.WriteLine(e.Status);
}
/* HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://localhost/api/");
request.CookieContainer = Cookie; // use the global cookie variable
string postData = "100";
byte[] data = Encoding.UTF8.GetBytes(postData);
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded; charset=utf-8";
request.ContentLength = data.Length;
using (Stream stream = request.GetRequestStream())
{
stream.Write(data, 0, data.Length);
}
WebResponse response = (HttpWebResponse)request.GetResponse();
string responseString = new StreamReader(response.GetResponseStream()).ReadToEnd();*/
WebClient csrfmiddlewaretoken = new WebClient();
CookieContainer cookieJar = new CookieContainer();
var request1 = (HttpWebRequest)HttpWebRequest.Create("http://localhost/api/");
request1.CookieContainer = cookieJar;
var response1 = request1.GetResponse();
string baseSiteString = csrfmiddlewaretoken.DownloadString("http://localhost/api/");
// string csrfToken = Regex.Match(baseSiteString, "<meta name=\"csrf-token\" content=\"(.*?)\" />").Groups[1].Value;
// wc.Headers.Add("X-CSRF-Token", csrfToken);
csrfmiddlewaretoken.Headers.Add("X-Requested-With", "XMLHttpRequest");
using (HttpWebResponse response2 = (HttpWebResponse)request1.GetResponse())
{
foreach (Cookie cookie1 in response2.Cookies)
{
//string cookie = csrfmiddlewaretoken.ResponseHeaders[HttpResponseHeader.SetCookie];//(response as HttpWebResponse).Headers[HttpResponseHeader.SetCookie];
Console.WriteLine(baseSiteString);
//Console.WriteLine("CSRF Token: {0}", csrfToken);
//Console.WriteLine("Cookie: {0}", cookie);
//
csrfmiddlewaretoken.Headers.Add(HttpRequestHeader.ContentType, "application/json; charset=utf-8");
//wc.Headers.Add(HttpRequestHeader.Accept, "application/json, text/javascript, */*; q=0.01");
var csrf1 = cookie1.Value;
string ARG1 = ("100");
string ARG2 = ("5");
string ARG = ("ARG");
string csrfmiddlewaretoken1 =cookie1.Name +"="+cookie1.Value;
csrfmiddlewaretoken.Headers.Add(HttpRequestHeader.Cookie, csrfmiddlewaretoken1);
//string csrf = string.Join(cookie, ARG1, ARG2);
//string dataString =cookie;
// string dataString = #"{""user"":{""email"":"""+uEmail+#""",""password"":"""+uPassword+#"""}}";
string dataString = "{\"ARG1\": \"100\", \"ARG2\": \"5\"}";
// string dataString = "csrftokenmiddlewaretoken="+csrfmiddlewaretoken;
//dataString += "&ARG1=10";
//dataString += "&ARG2=10";
byte[] dataBytes = Encoding.ASCII.GetBytes(dataString);
//byte[] respon2 = csrfmiddlewaretoken.DownloadData(new Uri("http://localhost/api/statusfunction/"));
WebRequest request = WebRequest.Create("http://localhost/api/statusfunction/?ARG");
// Get the response.
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
// Get the stream containing content returned by the server.
Stream dataStream = response.GetResponseStream();
// Open the stream using a StreamReader for easy access.
StreamReader reader = new StreamReader(dataStream);
// Read the content.
string responseFromServer = reader.ReadToEnd();
// Display the content.
Console.WriteLine(responseFromServer);
byte[] responseBytes = csrfmiddlewaretoken.UploadData(new Uri("http://localhost/api/multiplyfunction/"), "POST", dataBytes);
string responseString = Encoding.UTF8.GetString(responseBytes);
Console.WriteLine(responseString);
// byte[] res = csrfmiddlewaretoken.UploadFile("http://localhost/api/multiplyfunction/", #"ARG1:100");
// Console.WriteLine(result);
Console.WriteLine("value=" + cookie1.Value);
Console.WriteLine("name=" + cookie1.Name);
Console.WriteLine();
Console.Write((int)response2.StatusCode);
// PostRespone("http://localhost/api/multiplyfunction/");
}
}
}
}
}
Do you accept POST requests in your Django view? If yes, you should use #csrf_exempt decorator for the view that handles POST requests. The other option is to provide CSRF token from C# side.
I want to parse some html site like pluralsight,Forexample (https://app.pluralsight.com/id?), So How can I first login site programmaticaly (without using webbrowser control) then call another url (for example : Pluralsight) and get response and parse with Htmlagility pack.
But I've written a login code, but I do not know the next step.
public class Login
{
private CookieContainer Cookies = new CookieContainer();
public void SiteLogin(string username, string password)
{
Uri site = new Uri("https://app.pluralsight.com/id?");
HttpWebRequest wr = (HttpWebRequest)WebRequest.Create(site);
wr.Method = "Post";
wr.ContentType = "application/x-www-form-urlencoded";
wr.Referer = "https://app.pluralsight.com/id?";
wr.CookieContainer = Cookies;
var parameters = new Dictionary<string, string>{
{"realm", "vzw"},
{"goto",""},
{"gotoOnFail",""},
{"gx_charset", "UTF-8"},
{"rememberUserNameCheckBoxExists","Y"},
{"IDToken1", username},
{"IDToken2", password}
};
string input = string.Empty;
using (var requestStream = wr.GetRequestStream())
using (var writer = new StreamWriter(requestStream, Encoding.UTF8))
writer.Write(ParamsToFormEncoded(parameters));
using (var response = (HttpWebResponse)wr.GetResponse())
{
if (response.StatusCode == HttpStatusCode.OK)
{
//but I do not know the next step.
}
}
}
private string ParamsToFormEncoded(Dictionary<string, string> parameters)
{
return string.Join("&", parameters.Select(kvp => Uri.EscapeDataString(kvp.Key).Replace("%20", "+")
+ "=" + Uri.EscapeDataString(kvp.Value).Replace("20%", "+")
).ToArray());
}
}
You have to get the stream for the response via HttpWebResponse.GetResponseStream and then load the document via the Load method of the HtmlDocument.
var doc = new HtmlAgilityPack.HtmlDocument();
doc.Load(response.GetResponseStream());
//further processing...
http://www.conquerclub.com/game.php?game=13025037
Please does anyone know how I can programatically log onto this address and then get all the html of the page back as a string Using something like HttpWebRequest/HttpWebResponse.
username - "testuser1"
password - "testing"
(I've created an account on the site with these credentials so it will actually logon)
So far I've got this
private void toolStripButton1_Click(object sender, EventArgs e)
{
var request = WebRequest.Create("http://www.conquerclub.com/game.php?game=13025037");
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
StreamReader reader = new StreamReader(response.GetResponseStream());
richTextBox1.Text = reader.ReadToEnd();
}
This automatically re-directs and returns the html of the login page. How do I pass the username and password to login automatically so it retrieves the game page instead?
I know there are quite a few similar questions on this and I've spent several hours trying different things but can't get anything to work.
Edit:-
Looked into this further and this site uses cookies for logging in.
Tried this code but is still just returning the login page.
private void toolStripButton1_Click(object sender, EventArgs e)
{
string loginUri = "http://www.conquerclub.com/login.php";
string username = "testuser1";
string password = "testing";
string reqString = "username=" + username + "&password=" + password;
byte[] requestData = Encoding.UTF8.GetBytes(reqString);
CookieContainer cc = new CookieContainer();
var request = (HttpWebRequest)WebRequest.Create(loginUri);
request.Proxy = null;
request.AllowAutoRedirect = false;
request.CookieContainer = cc;
request.Method = "post";
request.ContentType = "application/x-www-form-urlencoded";
request.ContentLength = requestData.Length;
using (Stream s = request.GetRequestStream())
s.Write(requestData, 0, requestData.Length);
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
if (response.Cookies != null)
{
foreach (Cookie c in response.Cookies)
Console.WriteLine(c.Name + " = " + c.Value);
}
}
string newloginUri = "http://www.conquerclub.com/game.php?game=13025037";
HttpWebRequest newrequest = (HttpWebRequest)WebRequest.Create(newloginUri);
newrequest.Proxy = null;
newrequest.CookieContainer = cc;
using (HttpWebResponse newresponse = (HttpWebResponse)newrequest.GetResponse())
using (Stream resSteam = newresponse.GetResponseStream())
using (StreamReader sr = new StreamReader(resSteam))
richTextBox1.Text = sr.ReadToEnd();
}
Have found that the code at the bottom to log into the game page can be made to work by first logging in manually while using fiddler on firefox then copy and pasting the cookies and hard coding them into newrequest like this.
string newloginUri = "http://www.conquerclub.com/game.php?game=13025037";
HttpWebRequest newrequest = (HttpWebRequest)WebRequest.Create(newloginUri);
newrequest.Proxy = null;
newrequest.CookieContainer = new CookieContainer();
newrequest.CookieContainer.Add(new Uri("http://www.conquerclub.com"), new Cookie("PHPSESSID","86bte1ipainiq760vm2flv4h13"));
newrequest.CookieContainer.Add(new Uri("http://www.conquerclub.com"), new Cookie("phpbb3_jer7c_u", "648113"));
newrequest.CookieContainer.Add(new Uri("http://www.conquerclub.com"), new Cookie("phpbb3_jer7c_k", ""));
newrequest.CookieContainer.Add(new Uri("http://www.conquerclub.com"), new Cookie("phpbb3_jer7c_sid", "3eebb0771a68c4a58581e495c34b2c93"));
using (HttpWebResponse newresponse = (HttpWebResponse)newrequest.GetResponse())
using (Stream resSteam = newresponse.GetResponseStream())
using (StreamReader sr = new StreamReader(resSteam))
richTextBox1.Text = sr.ReadToEnd();
This returns the game page as I want it to but can't figure out how to get the login working so that it will return the right cookies. On debugging the code the cookies that come back are completely different to the ones I'm seeing in fiddler so looks like it's just not logging in.
The code that you've written does a GET of the URL given; however, to retrieve the content of the page when you log in you need to pretend that your WebRequest is actually filling in the form, by passing in all the form variables and issuing a POST request instead.
http://msdn.microsoft.com/en-us/library/debx8sh9.aspx gives a good walkthrough of the steps you require. The following isn't 100% complete, but should give you a step in the right direction:
var request = WebRequest.Create("http://www.conquerclub.com/login.php");
request.Method = "POST";
var parameters = new Dictionary<string, string> {
{ "username", "testuser1" },
{ "password", "testing" },
{ "redirect", "game.php?game=13025037" },
{ "direct", "63###www.conquerclub.com###" },
{ "protocol", "HTTP" },
{ "submit", "Login" } };
var content = string.Join( "&", ( from p in parameters select p.Key + "=" + HttpUtility.UrlEncode( p.Value) ).ToArray() ); ;
byte[] bytes = new byte[content.Length * sizeof(char)];
System.Buffer.BlockCopy(content.ToCharArray(), 0, bytes, 0, bytes.Length);
request.ContentLength = bytes.Length;
request.ContentType = "application/x-www-form-urlencoded";
Stream dataStream = request.GetRequestStream();
dataStream.Write( bytes, 0, bytes.Length );
dataStream.Close();
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
StreamReader reader = new StreamReader(response.GetResponseStream());
string result = reader.ReadToEnd();
Note that content includes various values that aren't visible on-screen; these can be seen by viewing the form's source, or (easier) by using the 'Network' tab on your browser debug toolbar of choice and monitoring the data being sent when the form is submitted.
You need to mimic the process of logging in. Look at the HTML code for the login page, and find out three things:
The <form> that sends the request, specifically the target (found in the attribute action) of the form.
Find out the method of sending the data (found in the attribute method)
Find the ID of the fields that represent the user name and password.
Having those you can easily construct a WebRequest that mimics a login.
Note, that this assumes a straight forward login screen (no AJAX, no CAPTCHAs).
I am making a call to a page on my site using webclient. I'm trying to get the result of the webpage put into a pdf so I am trying to get a string representation of the rendered page. The problem is that the request is not authenticated so all I get is a login screen. I have sent the UseDefaultCredentials property to true but I still get the same result. Below is a portion of my code:
WebClient webClient = new WebClient();
webClient.Encoding = Encoding.UTF8;
webClient.UseDefaultCredentials = true;
return Encoding.UTF8.GetString(webClient.UploadValues(link, "POST",form));
You need to give the WebClient object the credentials. Something like this...
WebClient client = new WebClient();
client.Credentials = new NetworkCredential("username", "password");
What kind of authentication are you using? If it's Forms authentication, then at best, you'll have to find the .ASPXAUTH cookie and pass it in the WebClient request.
At worst, it won't work.
Public Function getWeb(ByRef sURL As String) As String
Dim myWebClient As New System.Net.WebClient()
Try
Dim myCredentialCache As New System.Net.CredentialCache()
Dim myURI As New Uri(sURL)
myCredentialCache.Add(myURI, "ntlm", System.Net.CredentialCache.DefaultNetworkCredentials)
myWebClient.Encoding = System.Text.Encoding.UTF8
myWebClient.Credentials = myCredentialCache
Return myWebClient.DownloadString(myURI)
Catch ex As Exception
Return "Exception " & ex.ToString()
End Try
End Function
This helped me to call API that was using cookie authentication. I have passed authorization in header like this:
request.Headers.Set("Authorization", Utility.Helper.ReadCookie("AuthCookie"));
complete code:
// utility method to read the cookie value:
public static string ReadCookie(string cookieName)
{
var cookies = HttpContext.Current.Request.Cookies;
var cookie = cookies.Get(cookieName);
if (cookie != null)
return cookie.Value;
return null;
}
// using statements where you are creating your webclient
using System.Web.Script.Serialization;
using System.Net;
using System.IO;
// WebClient:
var requestUrl = "<API_url>";
var postRequest = new ClassRoom { name = "kushal seth" };
using (var webClient = new WebClient()) {
JavaScriptSerializer serializer = new JavaScriptSerializer();
byte[] requestData = Encoding.ASCII.GetBytes(serializer.Serialize(postRequest));
HttpWebRequest request = WebRequest.Create(requestUrl) as HttpWebRequest;
request.Method = "POST";
request.ContentType = "application/json";
request.ContentLength = requestData.Length;
request.ContentType = "application/json";
request.Expect = "application/json";
request.Headers.Set("Authorization", Utility.Helper.ReadCookie("AuthCookie"));
request.GetRequestStream().Write(requestData, 0, requestData.Length);
using (var response = (HttpWebResponse)request.GetResponse()) {
var reader = new StreamReader(response.GetResponseStream());
var objText = reader.ReadToEnd(); // objText will have the value
}
}