We Keep Coding

C# WebRequest - WebResponse to POST a webform with random generated token

I'm trying to login to www.autoscout24.de and retrieve adds and messages. Login form has a random generated hidden input/token. Being new to C#, I've read different tuts about using C# to login to websites and all I found was simple codes that work only in simple login forms (user:pass). I've imagined a 2-step approach: first make a GET request to retrieve needed data and a POST request with login credentials and other needed imputes. Using HtmlAgilityPack I'm passed first step but the second request just returns the login page again instead of "My account" page.
My code:
using System;
using System.IO;
using System.Net;
using System.Text;
namespace WebRequest__custom
{
class Program
{
static void Main(string[] args)
{
CookieContainer _cookies;
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("https://angebot.autoscout24.de/login?fromNavi=myAS24");
WebResponse _response = request.GetResponse();
Stream stream = _response.GetResponseStream();
StreamReader strReader = new StreamReader(stream);
string _cookiesHeader = _response.Headers["Set-cookie"];
_cookies = request.CookieContainer;
string _content = strReader.ReadToEnd();
//Console.WriteLine(_content.Substring(0,500));
var _dom = new HtmlAgilityPack.HtmlDocument();
_dom.LoadHtml(_content);
// Get POST link
var _postLinkNode = _dom.DocumentNode.SelectSingleNode("//*[#id='loginForm']/div[3]/form");
var postLink = _postLinkNode.Attributes["action"].Value;
//Console.WriteLine(postLink);
//get Token
var _tokenNode = _dom.DocumentNode.SelectSingleNode("//*[#id='loginForm']/div[3]/form/input");
var token = _tokenNode.Attributes["value"].Value;
//Console.WriteLine(token);
// Start login request
HttpWebRequest requestLogin = (HttpWebRequest)WebRequest.Create("https://accounts.autoscout24.com"+ postLink);
requestLogin.ContentType = "application/x-www-form-urlencoded";
requestLogin.Method = "POST";
requestLogin.KeepAlive = true;
requestLogin.AllowAutoRedirect = true;
string postData = "&__RequestVerificationToken=" + token;
postData += "&Username=web-cppxt#mail-tester.com";
postData += "&Password=Qwert123!";
postData += "&RememberMeCheckBox=on&RememberMe=true";
byte[] _bytes = Encoding.UTF8.GetBytes(postData);
requestLogin.ContentLength = _bytes.Length;
requestLogin.CookieContainer = _cookies;
using(Stream sr = requestLogin.GetRequestStream())
{
sr.Write(_bytes, 0, _bytes.Length);
}
WebResponse loginResponse = requestLogin.GetResponse();
StreamReader loginStreamReader = new StreamReader(loginResponse.GetResponseStream());
string secondPage = loginStreamReader.ReadToEnd();
Console.WriteLine(secondPage.Substring(0,500));
Console.ReadKey();
}
}
}

How to download using cefsharp in winforms [duplicate]

I am making a call to a page on my site using webclient. I'm trying to get the result of the webpage put into a pdf so I am trying to get a string representation of the rendered page. The problem is that the request is not authenticated so all I get is a login screen. I have sent the UseDefaultCredentials property to true but I still get the same result. Below is a portion of my code:
WebClient webClient = new WebClient();
webClient.Encoding = Encoding.UTF8;
webClient.UseDefaultCredentials = true;
return Encoding.UTF8.GetString(webClient.UploadValues(link, "POST",form));

You need to give the WebClient object the credentials. Something like this...
WebClient client = new WebClient();
client.Credentials = new NetworkCredential("username", "password");

What kind of authentication are you using? If it's Forms authentication, then at best, you'll have to find the .ASPXAUTH cookie and pass it in the WebClient request.
At worst, it won't work.

Public Function getWeb(ByRef sURL As String) As String
Dim myWebClient As New System.Net.WebClient()
Try
Dim myCredentialCache As New System.Net.CredentialCache()
Dim myURI As New Uri(sURL)
myCredentialCache.Add(myURI, "ntlm", System.Net.CredentialCache.DefaultNetworkCredentials)
myWebClient.Encoding = System.Text.Encoding.UTF8
myWebClient.Credentials = myCredentialCache
Return myWebClient.DownloadString(myURI)
Catch ex As Exception
Return "Exception " & ex.ToString()
End Try
End Function

This helped me to call API that was using cookie authentication. I have passed authorization in header like this:
request.Headers.Set("Authorization", Utility.Helper.ReadCookie("AuthCookie"));
complete code:
// utility method to read the cookie value:
public static string ReadCookie(string cookieName)
{
var cookies = HttpContext.Current.Request.Cookies;
var cookie = cookies.Get(cookieName);
if (cookie != null)
return cookie.Value;
return null;
}
// using statements where you are creating your webclient
using System.Web.Script.Serialization;
using System.Net;
using System.IO;
// WebClient:
var requestUrl = "<API_url>";
var postRequest = new ClassRoom { name = "kushal seth" };
using (var webClient = new WebClient()) {
JavaScriptSerializer serializer = new JavaScriptSerializer();
byte[] requestData = Encoding.ASCII.GetBytes(serializer.Serialize(postRequest));
HttpWebRequest request = WebRequest.Create(requestUrl) as HttpWebRequest;
request.Method = "POST";
request.ContentType = "application/json";
request.ContentLength = requestData.Length;
request.ContentType = "application/json";
request.Expect = "application/json";
request.Headers.Set("Authorization", Utility.Helper.ReadCookie("AuthCookie"));
request.GetRequestStream().Write(requestData, 0, requestData.Length);
using (var response = (HttpWebResponse)request.GetResponse()) {
var reader = new StreamReader(response.GetResponseStream());
var objText = reader.ReadToEnd(); // objText will have the value
}
}

Why am I getting this 401 unauthorized error while using Hammock with the Tumblr API to retrieve likes?

Here is the code that I am using:
public static void FetchXML()
{
_url = new Uri("http://api.tumblr.com/v2/blog/" + _username + ".tumblr.com/likes?api_key=REc3Z6l4ZYss11a8lX6KKje0X8Hsi9U77SyaPbQrOBBCGJGA6D");
var client = new RestClient();
client.Authority = _url.ToString();
var request = new RestRequest();
request.AddParameter("limit", "20");
request.AddParameter("offset", _offset.ToString());
var response = client.Request(request);
var content = response.Content.ToString();
var parsedResponse = JsonParser.FromJson(content);
}
If I take the Uri value and paste it into my browser (using a valid Tumblr username) I'm getting the correct Json, but in my application the content of response is:
"{\"meta\":{\"status\":401,\"msg\":\"Unauthorized\"},\"response\":[]}"
Anyone have any idea why this is? According to the Tumblr API
retrieving likes should only need the API key, which I am providing.

Hi you can use the below code to get the response.
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(URL);
request.Method = "GET";
request.ContentType = "Application/json";
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Stream receive = response.GetResponseStream();
StreamReader reader = new StreamReader(receive, Encoding.UTF8);
string respond = reader.ReadToEnd();

Forms authentication with NetworkCredentials fails

This is what I use to authenticate myself and make an action:
var uri = new Uri(#"http://localhost:5898/forums/AddPost/1");
var request = WebRequest.Create(uri);
request.Credentials = new CredentialCache { { new Uri(#"http://localhost:5898/Account/Login"), "Basic", new NetworkCredential("asdf", "ghijkl") } };
request.Method = "POST";
const string strPost = "Content=TestAplikacji&AddedValue=0";
StreamWriter myWriter = null;
request.ContentLength = strPost.Length;
request.ContentType = "application/x-www-form-urlencoded";
string response;
try
{
myWriter = new StreamWriter(request.GetRequestStream());
myWriter.Write(strPost);
}
catch
{
}
finally
{
if (myWriter != null) myWriter.Close();
}
var objResponse = (HttpWebResponse)request.GetResponse();
using (var sr = new StreamReader(objResponse.GetResponseStream()))
{
response = sr.ReadToEnd();
sr.Close();
}
When the code executes, in the response I get validation errors for login and password. But my provided login and password ("asdf", "ghijkl") are satisfying those requirements, and that means, the http://localhost:5898/Account/Login is not receiving credentials that I have provided. What am I doing wrong?

I believe, you should use Windows authentication / Basic Authentication, if you want to use the Network credential.
For forms authentication you need to post the username and password to the page just like what a browser will do and read the authentication cookie from the response. You should be using this cookie for communicating further with authenticated pages

How to add cookies to WebRequest?

I am trying to unit test some code, and I need to to replace this:
HttpWebRequest httpWebRequest = (HttpWebRequest)WebRequest.Create( uri );
httpWebRequest.CookieContainer = new CookieContainer();
with
WebRequest webRequest = WebRequest.Create( uri );
webRequest.CookieContainer = new CookieContainer();
Basically, how do I get cookies into the request without using a HttpWebRequest?

Based on your comments, you might consider writing an extension method:
public static bool TryAddCookie(this WebRequest webRequest, Cookie cookie)
{
HttpWebRequest httpRequest = webRequest as HttpWebRequest;
if (httpRequest == null)
{
return false;
}
if (httpRequest.CookieContainer == null)
{
httpRequest.CookieContainer = new CookieContainer();
}
httpRequest.CookieContainer.Add(cookie);
return true;
}
Then you can have code like:
WebRequest webRequest = WebRequest.Create( uri );
webRequest.TryAddCookie(new Cookie("someName","someValue"));

Try with something like this:
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://www.contoso.com/default.html");
request.CookieContainer = new CookieContainer();
request.CookieContainer.Add(new Cookie("ConstoCookie", "Chocolate Flavour"));

WebRequest is an abstract class that does not have a CookieContainer property. In addition you can't use the Headers collection (not implemented exception) so any attempt like webRequest.Headers.Add("Cookie", "...") will fail.
Sorry, but you have no chance to use cookies with WebRequest.
Stick on HttpWebRequest and add/edit as many cookies you like using its Headers collection!

dlev's answer ended up working, but I had problems implementing the solution ("The parameter '{0}' cannot be an empty string."), so I decided to write the full code in case anybody else has similar problems.
My goal was to get the html as a string, but I needed to add the cookies to the web request. This is the function that downloads the string using the cookies:
public static string DownloadString(string url, Encoding encoding, IDictionary<string, string> cookieNameValues)
{
using (var webClient = new WebClient())
{
var uri = new Uri(url);
var webRequest = WebRequest.Create(uri);
foreach(var nameValue in cookieNameValues)
{
webRequest.TryAddCookie(new Cookie(nameValue.Key, nameValue.Value, "/", uri.Host));
}
var response = webRequest.GetResponse();
var receiveStream = response.GetResponseStream();
var readStream = new StreamReader(receiveStream, encoding);
var htmlCode = readStream.ReadToEnd();
return htmlCode;
}
}
We are using the code from dlev's answer:
public static bool TryAddCookie(this WebRequest webRequest, Cookie cookie)
{
HttpWebRequest httpRequest = webRequest as HttpWebRequest;
if (httpRequest == null)
{
return false;
}
if (httpRequest.CookieContainer == null)
{
httpRequest.CookieContainer = new CookieContainer();
}
httpRequest.CookieContainer.Add(cookie);
return true;
}
This is how you use the full code:
var cookieNameValues = new Dictionary<string, string>();
cookieNameValues.Add("varName", "varValue");
var htmlResult = DownloadString(url, Encoding.UTF8, cookieNameValues);