We Keep Coding

C# WebRequest - WebResponse to POST a webform with random generated token

I'm trying to login to www.autoscout24.de and retrieve adds and messages. Login form has a random generated hidden input/token. Being new to C#, I've read different tuts about using C# to login to websites and all I found was simple codes that work only in simple login forms (user:pass). I've imagined a 2-step approach: first make a GET request to retrieve needed data and a POST request with login credentials and other needed imputes. Using HtmlAgilityPack I'm passed first step but the second request just returns the login page again instead of "My account" page.
My code:
using System;
using System.IO;
using System.Net;
using System.Text;
namespace WebRequest__custom
{
class Program
{
static void Main(string[] args)
{
CookieContainer _cookies;
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("https://angebot.autoscout24.de/login?fromNavi=myAS24");
WebResponse _response = request.GetResponse();
Stream stream = _response.GetResponseStream();
StreamReader strReader = new StreamReader(stream);
string _cookiesHeader = _response.Headers["Set-cookie"];
_cookies = request.CookieContainer;
string _content = strReader.ReadToEnd();
//Console.WriteLine(_content.Substring(0,500));
var _dom = new HtmlAgilityPack.HtmlDocument();
_dom.LoadHtml(_content);
// Get POST link
var _postLinkNode = _dom.DocumentNode.SelectSingleNode("//*[#id='loginForm']/div[3]/form");
var postLink = _postLinkNode.Attributes["action"].Value;
//Console.WriteLine(postLink);
//get Token
var _tokenNode = _dom.DocumentNode.SelectSingleNode("//*[#id='loginForm']/div[3]/form/input");
var token = _tokenNode.Attributes["value"].Value;
//Console.WriteLine(token);
// Start login request
HttpWebRequest requestLogin = (HttpWebRequest)WebRequest.Create("https://accounts.autoscout24.com"+ postLink);
requestLogin.ContentType = "application/x-www-form-urlencoded";
requestLogin.Method = "POST";
requestLogin.KeepAlive = true;
requestLogin.AllowAutoRedirect = true;
string postData = "&__RequestVerificationToken=" + token;
postData += "&Username=web-cppxt#mail-tester.com";
postData += "&Password=Qwert123!";
postData += "&RememberMeCheckBox=on&RememberMe=true";
byte[] _bytes = Encoding.UTF8.GetBytes(postData);
requestLogin.ContentLength = _bytes.Length;
requestLogin.CookieContainer = _cookies;
using(Stream sr = requestLogin.GetRequestStream())
{
sr.Write(_bytes, 0, _bytes.Length);
}
WebResponse loginResponse = requestLogin.GetResponse();
StreamReader loginStreamReader = new StreamReader(loginResponse.GetResponseStream());
string secondPage = loginStreamReader.ReadToEnd();
Console.WriteLine(secondPage.Substring(0,500));
Console.ReadKey();
}
}
}

Too many automatic redirects have been attempted in WebClient

I need to allow redirection in my WebClient 'cause I've this url:
http://int.soccerway.com/national/algeria/ligue-2/c207/
when I add this in the browser I'll get a redirect to this:
http://int.soccerway.com/national/algeria/ligue-2/20172018/regular-season/r43168/
when I perform a request I execute this method:
public static string GetData(string url)
{
HttpWebRequest webReq = (HttpWebRequest)HttpWebRequest.Create(url);
try
{
webReq.CookieContainer = new CookieContainer();
webReq.Method = "GET";
webReq.AllowAutoRedirect = true;
webReq.MaximumAutomaticRedirections = 1;
using (WebResponse response = webReq.GetResponse())
{
using (Stream stream = response.GetResponseStream())
{
StreamReader reader = new StreamReader(stream);
return reader.ReadToEnd();
}
}
...
}
but this throws an exception:
Too many automatic redirects have been attempted
other threads suggest to add a cookie container, I did so but the error happen occurs again. How can I solve this?

How to solve the task called 'Fast and Furious' from 'Break In 2017' challenge in C#?

I try to solve the task called Fast and Furious from Break In 2017 challenge.
The task is simple. Need to HTTP post back the result of a math expression contained in a HTML page. It can be achieved by this python script:
import requests
import re
url = 'https://felicity.iiit.ac.in/contest/extra/fastandfurious/'
s = requests.Session()
r = s.get(url)
print(r.text)
m = re.search('\(.*\)', r.text)
while m:
ans = eval(m[0])
print(m[0] + ' -> ' + str(ans))
r = s.post(url, data={'ques_ans' : ans})
print(r.text)
if ('the_flag_is_' in r.text): break
m = re.search('\(.*\)', r.text)
I want to do the same in C#. I tried like this:
using System;
using System.Data;
using System.IO;
using System.Net;
using System.Text;
using System.Text.RegularExpressions;
class Program
{
static CookieContainer cookies = new CookieContainer();
static HttpWebRequest Create(string url)
{
var request = (HttpWebRequest)WebRequest.Create(url);
request.CookieContainer = cookies;
request.Accept = "*/*";
request.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
request.UserAgent = "dummy";
return request;
}
static string Get(string url)
{
var request = Create(url);
request.Method = "GET";
using (var response = request.GetResponse())
using (var reader = new StreamReader(response.GetResponseStream()))
{
return reader.ReadToEnd();
}
}
static string Post(string url, string postData)
{
var request = Create(url);
request.Method = "POST";
var data = Encoding.UTF8.GetBytes(postData);
request.ContentType = "application/x-www-form-urlencoded";
request.ContentLength = data.Length;
using (var stream = request.GetRequestStream())
{
stream.Write(data, 0, data.Length);
}
using (var response = request.GetResponse())
using (var reader = new StreamReader(response.GetResponseStream()))
{
return reader.ReadToEnd();
}
}
static string Eval(string expression)
{
DataTable dt = new DataTable();
return dt.Compute(expression, "").ToString();
}
static void Main()
{
string url = "https://felicity.iiit.ac.in/contest/extra/fastandfurious";
string text = Get(url);
Console.WriteLine(text);
var m = Regex.Match(text, #"\(.*\)");
while (m.Success)
{
var ans = Eval(m.Value);
Console.WriteLine(m.Value + " -> " + ans);
text = Post(url, "ques_ans=" + ans);
Console.WriteLine(text);
if (text.Contains("the_flag_is_")) break;
m = Regex.Match(text, #"\(.*\)");
}
}
}
But it does not work, because I always get back the 'Level 1' question. I used HttpWebRequest.CookieContainer property to reuse the cookies across different requests to keep up a session.
I don't know what is the problem. Maybe the session doesn't work.
Or perhaps HttpWebRequest is too slow to post back the result in time.
How to solve this automation task in C#?

Your code isn't handling being redirected. Watching the traffic we can see that this particular server would like your requests to /fastandfurious to end with a trailing slash. so change '../fastandfurious' to '../fastandfurious/' and that will fix it.

Python Requests Post Failing With 500 Error

I am using this excellent project on GitHub (https://github.com/cjyoung/MouseBitesWPF). However, that is written in C# and I really need something written in Python. I have boiled that code down to the absolute bare bones of what it needs to work and came up with this:
using System;
using System.Collections.Generic;
using System.Collections.ObjectModel;
using System.ComponentModel;
using System.IO;
using System.Linq;
using System.Net;
using System.Text;
using System.Text.RegularExpressions;
using System.Threading.Tasks;
namespace DisneyFinderSmaller
{
class Program
{
static CookieContainer cookieJar = new CookieContainer();
static internal string rootUrl = "https://disneyworld.disney.go.com";
static internal string siteUrl = "/dining/";
static internal string diningSearchUrl = "/finder/dining-availability";
static void Main(string[] args)
{
LaunchSearchInstance();
string test = Console.ReadLine();
}
private static void LaunchSearchInstance()
{
string result = getCookiesFromRequest(rootUrl + siteUrl, "", "GET");
string pep_csrf = "";
Match match = Regex.Match(result, "<input[^>]*name=['\"]pep_csrf['\"][^>]*value=['\"]([^'\"]*)['\"]*[^>]>", RegexOptions.Singleline & RegexOptions.IgnoreCase);
pep_csrf = match.Groups[1].ToString();
ConductSearch(pep_csrf);
}
private static void ConductSearch(string pep_csrf)
{
string postString = string.Format("&searchDate={1}" +
"&skipPricing=true" +
"&searchTime={2}" +
"&partySize={3}" +
"&id={0}%3BentityType%3Drestaurant" +
"&type=dining" +
"&pep_csrf={4}",
"293704",
"2015-11-18",
"80000714",
"2",
pep_csrf);
string result = getCookiesFromRequest(rootUrl + diningSearchUrl, postString, "POST");
System.Console.WriteLine(result);
}
private static String getCookiesFromRequest(string url, string postString, string method = "POST")
{
String result = "";
byte[] postBytes = Encoding.ASCII.GetBytes(postString);
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.Method = method;
request.Referer = rootUrl + siteUrl;
request.CookieContainer = cookieJar;
if (method == "POST")
{
request.ContentType = "application/x-www-form-urlencoded";
Stream postStream = request.GetRequestStream();
postStream.Write(postBytes, 0, postBytes.Length);
postStream.Close();
}
try
{
HttpWebResponse webResponse = (HttpWebResponse)request.GetResponse();
Stream responseStream = webResponse.GetResponseStream();
StreamReader responseStreamReader = new StreamReader(responseStream);
result = responseStreamReader.ReadToEnd();
responseStream.Close();
webResponse.Close();
}
catch (Exception ex)
{
Console.WriteLine("IOException source: {0}", ex.Source);
}
return result;
}
}
}
In my efforts to translate this to Python using Requests, I have come up with this:
#!/usr/bin/env python
import requests
url = "https://disneyworld.disney.go.com/dining/"
url2 = "https://disneyworld.disney.go.com/dining/finder/dining-availability"
session = requests.Session()
tokenRequest = session.get(url, headers=header)
start = tokenRequest.content.find('''id="pep_csrf"''')
pep = tokenRequest.content[start+21:tokenRequest.content.find('>',start+22)-1]
raw = "&searchDate=2015-11-18&skipPricing=true&searchTime=80000714&partySize=2&id=293704%3BentityType%3Drestaurant&type=dining&pep_csrf=" + pep
headers = {
'Content-Type': 'application/x-www-form-urlencoded',
'referer': 'https://disneyworld.disney.go.com/dining/',
'method' : 'POST'
}
result = session.post(url2, data=raw, headers=headers)
print result.status_code
But this doesn't work and returns a status code of 500.
Any thoughts on where things are going wrong? I have been hanging my head against the wall for a few days and any insight at all would be so appreciated.

NULL JSON posted onto PHP server using HTTP requests from C#

I'm trying to post a JSON string on a PHP page using HTTP response methods as follows:
using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Drawing;
using System.Linq;
using System.Text;
using System.Windows.Forms;
using System.Net;
using System.IO;
using System.Web.Script.Serialization;
using System.Web;
namespace http_requests
{
public partial class Form1 : Form
{
public Form1()
{
InitializeComponent();
}
private void button1_Click(object sender, EventArgs e)
{
//var httpWebRequest = (HttpWebRequest)WebRequest.Create("http://localhost/abc/products.php");
//httpWebRequest.ContentType = "application/json";
//httpWebRequest.Method = "POST";
//using (var streamWriter = new StreamWriter(httpWebRequest.GetRequestStream()))
//{
// string json = new JavaScriptSerializer().Serialize(new
// {
// user = "Foo",
// password = "Baz"
// });
// streamWriter.Write(json);
// streamWriter.Flush();
// streamWriter.Close();
// var httpResponse = (HttpWebResponse)httpWebRequest.GetResponse();
// using (var streamReader = new StreamReader(httpResponse.GetResponseStream()))
// {
// var result = streamReader.ReadToEnd();
// }
//}
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create("http://localhost/ABC/products.php");
request.Method = WebRequestMethods.Http.Post;
string DataToPost = new JavaScriptSerializer().Serialize(new
{
user = "Foo",
password = "Baz"
});
byte[] bytes = Encoding.UTF8.GetBytes(DataToPost);
string byteString = Encoding.UTF8.GetString(bytes);
Stream os = null;
//string postData = "firstName=" + HttpUtility.UrlEncode(p.firstName) +
request.ContentLength = bytes.Length;
request.ContentType = "application/x-www-form-urlencoded";
os = request.GetRequestStream();
os.Write(bytes, 0, bytes.Length);
//StreamWriter writer = new StreamWriter(request.GetRequestStream());
//writer.Write(DataToPost);
//writer.Close();
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
//StreamReader reader = new StreamReader(response.GetResponseStream());
using (var streamReader = new StreamReader(response.GetResponseStream()))
{
var result = streamReader.ReadToEnd();
richTextBox1.AppendText("R : " + result);
Console.WriteLine(streamReader.ReadToEnd().Trim());
}
//richTextBox1.Text = response.ToString();
}
}
}
I tried it in many different ways (converting to bytes too) but still posts a NULL array.
PHP Code:
<?php
$json = $_POST;
if (isset($json)) {
echo "This var is set so I will print.";
//var_dump($json);
var_dump(json_decode(file_get_contents('php://input')));
}
?>
When I try to get tha response from server and print onto a text box, it prints right:
R : This var is set so I will print.object(stdClass)#1 (2) {
["user"]=>
string(3) "Foo"
["password"]=>
string(3) "Baz"
}
but i'm unable to check it on my PHP page, it says:
This var is set so I will print.NULL
Not sure if its posting a JSON onto PHP or not, but it sure does posts a NULL.
I want to see the JSON on PHP page, Any help would be appreciated.
Thank you,
Revathy

There is nothing wrong with your c# client side code, the problem is that visiting a site in your browser is a seperate request from the c# post, so you wont see anything.
As per my comment, if you want to see the data in a browser after a post i c#, you will need to save and retrieve it.
Here is a simple example using a text file to save post data and display it:
//if post request
if($_SERVER['REQUEST_METHOD']=='POST'){
//get data from POST
$data = file_get_contents('php://input');
//save to file
file_put_contents('data.txt', $data);
die('Saved');
}
//else if its a get request (eg view in browser)
var_dump(json_decode(file_get_contents('data.txt')));