We Keep Coding

C# WebRequest - WebResponse to POST a webform with random generated token

I'm trying to login to www.autoscout24.de and retrieve adds and messages. Login form has a random generated hidden input/token. Being new to C#, I've read different tuts about using C# to login to websites and all I found was simple codes that work only in simple login forms (user:pass). I've imagined a 2-step approach: first make a GET request to retrieve needed data and a POST request with login credentials and other needed imputes. Using HtmlAgilityPack I'm passed first step but the second request just returns the login page again instead of "My account" page.
My code:
using System;
using System.IO;
using System.Net;
using System.Text;
namespace WebRequest__custom
{
class Program
{
static void Main(string[] args)
{
CookieContainer _cookies;
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("https://angebot.autoscout24.de/login?fromNavi=myAS24");
WebResponse _response = request.GetResponse();
Stream stream = _response.GetResponseStream();
StreamReader strReader = new StreamReader(stream);
string _cookiesHeader = _response.Headers["Set-cookie"];
_cookies = request.CookieContainer;
string _content = strReader.ReadToEnd();
//Console.WriteLine(_content.Substring(0,500));
var _dom = new HtmlAgilityPack.HtmlDocument();
_dom.LoadHtml(_content);
// Get POST link
var _postLinkNode = _dom.DocumentNode.SelectSingleNode("//*[#id='loginForm']/div[3]/form");
var postLink = _postLinkNode.Attributes["action"].Value;
//Console.WriteLine(postLink);
//get Token
var _tokenNode = _dom.DocumentNode.SelectSingleNode("//*[#id='loginForm']/div[3]/form/input");
var token = _tokenNode.Attributes["value"].Value;
//Console.WriteLine(token);
// Start login request
HttpWebRequest requestLogin = (HttpWebRequest)WebRequest.Create("https://accounts.autoscout24.com"+ postLink);
requestLogin.ContentType = "application/x-www-form-urlencoded";
requestLogin.Method = "POST";
requestLogin.KeepAlive = true;
requestLogin.AllowAutoRedirect = true;
string postData = "&__RequestVerificationToken=" + token;
postData += "&Username=web-cppxt#mail-tester.com";
postData += "&Password=Qwert123!";
postData += "&RememberMeCheckBox=on&RememberMe=true";
byte[] _bytes = Encoding.UTF8.GetBytes(postData);
requestLogin.ContentLength = _bytes.Length;
requestLogin.CookieContainer = _cookies;
using(Stream sr = requestLogin.GetRequestStream())
{
sr.Write(_bytes, 0, _bytes.Length);
}
WebResponse loginResponse = requestLogin.GetResponse();
StreamReader loginStreamReader = new StreamReader(loginResponse.GetResponseStream());
string secondPage = loginStreamReader.ReadToEnd();
Console.WriteLine(secondPage.Substring(0,500));
Console.ReadKey();
}
}
}

How to deal with JavaScript when fetching http Response in C# using HttpWebRequest?

I have a C# code (it's a Web Application, that's hosted on IIS) where I use HttpWebRequest to get HttpWebResponse. There I make request to any website & get the response as string, then I analysis the response string. But recently I get the response where JavaScript performs data fetching after page is loaded in browser.
I tried to debug this in firebug & saw that at bottom of response there's a JavaScript function that updates the dom elements after pageload. Is there any way that I could do the same in my C# code. I have searched on net about this found not solution till now.
Following is the code I am using:
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
foreach (Cookie cook in response.Cookies)
{
Cookie cookie = new Cookie();
cookie.Name = cook.Name;
cookie.Value = cook.Value;
cookie.Domain = cook.Domain;
cookie.Expires = DateTime.Now.AddDays(10);
cookieList.Add(cookie);
}
string postData = string.Format("username=" + txtUserID.Text + "&password=" + txtPwd.Text + "&url=https://example.com/&game=");
byte[] postBytes = Encoding.UTF8.GetBytes(postData);
HttpWebRequest req = (HttpWebRequest)WebRequest.Create("https://login.example.com/Login/authenticate");
req.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:42.0) Gecko/20100101 Firefox/42.0";
req.KeepAlive = true;
req.AutomaticDecompression = DecompressionMethods.GZip;
////set the cookie
req.CookieContainer = new CookieContainer();
foreach (Cookie cook in cookieList)
{
Cookie cookie = new Cookie();
cookie.Name = cook.Name;
cookie.Value = cook.Value;
cookie.Domain = cook.Domain;
cookie.Expires = DateTime.Now.AddDays(10);
req.CookieContainer.Add(cookie);
}
req.Headers.Add("Accept-Encoding", "gzip, deflate");
req.Headers.Add("Accept-Language", "en-GB,en-US;q=0.8,en;q=0.6");//en-GB,en-US;q=0.8,en;q=0.6
req.Method = "POST";
req.Host = "login.example.com";
req.Referer = "https://login.example.com/Login/logout";
req.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8";
req.ContentType = "application/x-www-form-urlencoded;";
req.ContentLength = postBytes.Length;
//getting the request stream and posting data
StreamWriter requestwriter = new StreamWriter(req.GetRequestStream(), System.Text.Encoding.ASCII);
requestwriter.Write(postData);
requestwriter.Close();
HttpWebResponse myHttpWebResponse = (HttpWebResponse)req.GetResponse();
Stream responseStream = myHttpWebResponse.GetResponseStream();
StreamReader myStreamReader = new StreamReader(responseStream, Encoding.ASCII);
string responseString = myStreamReader.ReadToEnd();
myStreamReader.Close();
responseStream.Close();
myHttpWebResponse.Close();

I finally got the easy solution to my need. following is the link that I followed:
Link to tutorial
Following is the code that'll get the results:
First you'll need to import following:
using System.Drawing;
using OpenQA.Selenium;
using OpenQA.Selenium.PhantomJS;
using System.Text.RegularExpressions;
using System.IO;
using HtmlAgilityPack;
Now the code:
var options = new PhantomJSOptions();
var driver = new PhantomJSDriver(options);
driver.Manage().Window.Size = new Size(1360, 728);
var size = driver.Manage().Window.Size;
driver.Navigate().GoToUrl("https://example.com/");
string url = driver.Url;
//the driver can now provide you with what you need (it will execute the script)
//get the source of the page
var source = driver.PageSource;
//fully navigate the dom
var pathElement1 = driver.FindElementByName("username");
var pathElement2 = driver.FindElementByName("password");
var pathElement3 = driver.FindElementByXPath("//button[#class='SubmitButton']");
pathElement1.Clear();
pathElement1.SendKeys("username");
pathElement2.Clear();
pathElement2.SendKeys("password");
pathElement3.Click();
//Now get the response after login button click
source = driver.PageSource;

Forms authentication with NetworkCredentials fails

This is what I use to authenticate myself and make an action:
var uri = new Uri(#"http://localhost:5898/forums/AddPost/1");
var request = WebRequest.Create(uri);
request.Credentials = new CredentialCache { { new Uri(#"http://localhost:5898/Account/Login"), "Basic", new NetworkCredential("asdf", "ghijkl") } };
request.Method = "POST";
const string strPost = "Content=TestAplikacji&AddedValue=0";
StreamWriter myWriter = null;
request.ContentLength = strPost.Length;
request.ContentType = "application/x-www-form-urlencoded";
string response;
try
{
myWriter = new StreamWriter(request.GetRequestStream());
myWriter.Write(strPost);
}
catch
{
}
finally
{
if (myWriter != null) myWriter.Close();
}
var objResponse = (HttpWebResponse)request.GetResponse();
using (var sr = new StreamReader(objResponse.GetResponseStream()))
{
response = sr.ReadToEnd();
sr.Close();
}
When the code executes, in the response I get validation errors for login and password. But my provided login and password ("asdf", "ghijkl") are satisfying those requirements, and that means, the http://localhost:5898/Account/Login is not receiving credentials that I have provided. What am I doing wrong?

I believe, you should use Windows authentication / Basic Authentication, if you want to use the Network credential.
For forms authentication you need to post the username and password to the page just like what a browser will do and read the authentication cookie from the response. You should be using this cookie for communicating further with authenticated pages

How to add cookies to WebRequest?

I am trying to unit test some code, and I need to to replace this:
HttpWebRequest httpWebRequest = (HttpWebRequest)WebRequest.Create( uri );
httpWebRequest.CookieContainer = new CookieContainer();
with
WebRequest webRequest = WebRequest.Create( uri );
webRequest.CookieContainer = new CookieContainer();
Basically, how do I get cookies into the request without using a HttpWebRequest?

Based on your comments, you might consider writing an extension method:
public static bool TryAddCookie(this WebRequest webRequest, Cookie cookie)
{
HttpWebRequest httpRequest = webRequest as HttpWebRequest;
if (httpRequest == null)
{
return false;
}
if (httpRequest.CookieContainer == null)
{
httpRequest.CookieContainer = new CookieContainer();
}
httpRequest.CookieContainer.Add(cookie);
return true;
}
Then you can have code like:
WebRequest webRequest = WebRequest.Create( uri );
webRequest.TryAddCookie(new Cookie("someName","someValue"));

Try with something like this:
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://www.contoso.com/default.html");
request.CookieContainer = new CookieContainer();
request.CookieContainer.Add(new Cookie("ConstoCookie", "Chocolate Flavour"));

WebRequest is an abstract class that does not have a CookieContainer property. In addition you can't use the Headers collection (not implemented exception) so any attempt like webRequest.Headers.Add("Cookie", "...") will fail.
Sorry, but you have no chance to use cookies with WebRequest.
Stick on HttpWebRequest and add/edit as many cookies you like using its Headers collection!

dlev's answer ended up working, but I had problems implementing the solution ("The parameter '{0}' cannot be an empty string."), so I decided to write the full code in case anybody else has similar problems.
My goal was to get the html as a string, but I needed to add the cookies to the web request. This is the function that downloads the string using the cookies:
public static string DownloadString(string url, Encoding encoding, IDictionary<string, string> cookieNameValues)
{
using (var webClient = new WebClient())
{
var uri = new Uri(url);
var webRequest = WebRequest.Create(uri);
foreach(var nameValue in cookieNameValues)
{
webRequest.TryAddCookie(new Cookie(nameValue.Key, nameValue.Value, "/", uri.Host));
}
var response = webRequest.GetResponse();
var receiveStream = response.GetResponseStream();
var readStream = new StreamReader(receiveStream, encoding);
var htmlCode = readStream.ReadToEnd();
return htmlCode;
}
}
We are using the code from dlev's answer:
public static bool TryAddCookie(this WebRequest webRequest, Cookie cookie)
{
HttpWebRequest httpRequest = webRequest as HttpWebRequest;
if (httpRequest == null)
{
return false;
}
if (httpRequest.CookieContainer == null)
{
httpRequest.CookieContainer = new CookieContainer();
}
httpRequest.CookieContainer.Add(cookie);
return true;
}
This is how you use the full code:
var cookieNameValues = new Dictionary<string, string>();
cookieNameValues.Add("varName", "varValue");
var htmlResult = DownloadString(url, Encoding.UTF8, cookieNameValues);

How do I authenticate a WebClient request?

I am making a call to a page on my site using webclient. I'm trying to get the result of the webpage put into a pdf so I am trying to get a string representation of the rendered page. The problem is that the request is not authenticated so all I get is a login screen. I have sent the UseDefaultCredentials property to true but I still get the same result. Below is a portion of my code:
WebClient webClient = new WebClient();
webClient.Encoding = Encoding.UTF8;
webClient.UseDefaultCredentials = true;
return Encoding.UTF8.GetString(webClient.UploadValues(link, "POST",form));

You need to give the WebClient object the credentials. Something like this...
WebClient client = new WebClient();
client.Credentials = new NetworkCredential("username", "password");

What kind of authentication are you using? If it's Forms authentication, then at best, you'll have to find the .ASPXAUTH cookie and pass it in the WebClient request.
At worst, it won't work.

Public Function getWeb(ByRef sURL As String) As String
Dim myWebClient As New System.Net.WebClient()
Try
Dim myCredentialCache As New System.Net.CredentialCache()
Dim myURI As New Uri(sURL)
myCredentialCache.Add(myURI, "ntlm", System.Net.CredentialCache.DefaultNetworkCredentials)
myWebClient.Encoding = System.Text.Encoding.UTF8
myWebClient.Credentials = myCredentialCache
Return myWebClient.DownloadString(myURI)
Catch ex As Exception
Return "Exception " & ex.ToString()
End Try
End Function

This helped me to call API that was using cookie authentication. I have passed authorization in header like this:
request.Headers.Set("Authorization", Utility.Helper.ReadCookie("AuthCookie"));
complete code:
// utility method to read the cookie value:
public static string ReadCookie(string cookieName)
{
var cookies = HttpContext.Current.Request.Cookies;
var cookie = cookies.Get(cookieName);
if (cookie != null)
return cookie.Value;
return null;
}
// using statements where you are creating your webclient
using System.Web.Script.Serialization;
using System.Net;
using System.IO;
// WebClient:
var requestUrl = "<API_url>";
var postRequest = new ClassRoom { name = "kushal seth" };
using (var webClient = new WebClient()) {
JavaScriptSerializer serializer = new JavaScriptSerializer();
byte[] requestData = Encoding.ASCII.GetBytes(serializer.Serialize(postRequest));
HttpWebRequest request = WebRequest.Create(requestUrl) as HttpWebRequest;
request.Method = "POST";
request.ContentType = "application/json";
request.ContentLength = requestData.Length;
request.ContentType = "application/json";
request.Expect = "application/json";
request.Headers.Set("Authorization", Utility.Helper.ReadCookie("AuthCookie"));
request.GetRequestStream().Write(requestData, 0, requestData.Length);
using (var response = (HttpWebResponse)request.GetResponse()) {
var reader = new StreamReader(response.GetResponseStream());
var objText = reader.ReadToEnd(); // objText will have the value
}
}