We Keep Coding

C# WebRequest - WebResponse to POST a webform with random generated token

I'm trying to login to www.autoscout24.de and retrieve adds and messages. Login form has a random generated hidden input/token. Being new to C#, I've read different tuts about using C# to login to websites and all I found was simple codes that work only in simple login forms (user:pass). I've imagined a 2-step approach: first make a GET request to retrieve needed data and a POST request with login credentials and other needed imputes. Using HtmlAgilityPack I'm passed first step but the second request just returns the login page again instead of "My account" page.
My code:
using System;
using System.IO;
using System.Net;
using System.Text;
namespace WebRequest__custom
{
class Program
{
static void Main(string[] args)
{
CookieContainer _cookies;
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("https://angebot.autoscout24.de/login?fromNavi=myAS24");
WebResponse _response = request.GetResponse();
Stream stream = _response.GetResponseStream();
StreamReader strReader = new StreamReader(stream);
string _cookiesHeader = _response.Headers["Set-cookie"];
_cookies = request.CookieContainer;
string _content = strReader.ReadToEnd();
//Console.WriteLine(_content.Substring(0,500));
var _dom = new HtmlAgilityPack.HtmlDocument();
_dom.LoadHtml(_content);
// Get POST link
var _postLinkNode = _dom.DocumentNode.SelectSingleNode("//*[#id='loginForm']/div[3]/form");
var postLink = _postLinkNode.Attributes["action"].Value;
//Console.WriteLine(postLink);
//get Token
var _tokenNode = _dom.DocumentNode.SelectSingleNode("//*[#id='loginForm']/div[3]/form/input");
var token = _tokenNode.Attributes["value"].Value;
//Console.WriteLine(token);
// Start login request
HttpWebRequest requestLogin = (HttpWebRequest)WebRequest.Create("https://accounts.autoscout24.com"+ postLink);
requestLogin.ContentType = "application/x-www-form-urlencoded";
requestLogin.Method = "POST";
requestLogin.KeepAlive = true;
requestLogin.AllowAutoRedirect = true;
string postData = "&__RequestVerificationToken=" + token;
postData += "&Username=web-cppxt#mail-tester.com";
postData += "&Password=Qwert123!";
postData += "&RememberMeCheckBox=on&RememberMe=true";
byte[] _bytes = Encoding.UTF8.GetBytes(postData);
requestLogin.ContentLength = _bytes.Length;
requestLogin.CookieContainer = _cookies;
using(Stream sr = requestLogin.GetRequestStream())
{
sr.Write(_bytes, 0, _bytes.Length);
}
WebResponse loginResponse = requestLogin.GetResponse();
StreamReader loginStreamReader = new StreamReader(loginResponse.GetResponseStream());
string secondPage = loginStreamReader.ReadToEnd();
Console.WriteLine(secondPage.Substring(0,500));
Console.ReadKey();
}
}
}

How to solve the task called 'Fast and Furious' from 'Break In 2017' challenge in C#?

I try to solve the task called Fast and Furious from Break In 2017 challenge.
The task is simple. Need to HTTP post back the result of a math expression contained in a HTML page. It can be achieved by this python script:
import requests
import re
url = 'https://felicity.iiit.ac.in/contest/extra/fastandfurious/'
s = requests.Session()
r = s.get(url)
print(r.text)
m = re.search('\(.*\)', r.text)
while m:
ans = eval(m[0])
print(m[0] + ' -> ' + str(ans))
r = s.post(url, data={'ques_ans' : ans})
print(r.text)
if ('the_flag_is_' in r.text): break
m = re.search('\(.*\)', r.text)
I want to do the same in C#. I tried like this:
using System;
using System.Data;
using System.IO;
using System.Net;
using System.Text;
using System.Text.RegularExpressions;
class Program
{
static CookieContainer cookies = new CookieContainer();
static HttpWebRequest Create(string url)
{
var request = (HttpWebRequest)WebRequest.Create(url);
request.CookieContainer = cookies;
request.Accept = "*/*";
request.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
request.UserAgent = "dummy";
return request;
}
static string Get(string url)
{
var request = Create(url);
request.Method = "GET";
using (var response = request.GetResponse())
using (var reader = new StreamReader(response.GetResponseStream()))
{
return reader.ReadToEnd();
}
}
static string Post(string url, string postData)
{
var request = Create(url);
request.Method = "POST";
var data = Encoding.UTF8.GetBytes(postData);
request.ContentType = "application/x-www-form-urlencoded";
request.ContentLength = data.Length;
using (var stream = request.GetRequestStream())
{
stream.Write(data, 0, data.Length);
}
using (var response = request.GetResponse())
using (var reader = new StreamReader(response.GetResponseStream()))
{
return reader.ReadToEnd();
}
}
static string Eval(string expression)
{
DataTable dt = new DataTable();
return dt.Compute(expression, "").ToString();
}
static void Main()
{
string url = "https://felicity.iiit.ac.in/contest/extra/fastandfurious";
string text = Get(url);
Console.WriteLine(text);
var m = Regex.Match(text, #"\(.*\)");
while (m.Success)
{
var ans = Eval(m.Value);
Console.WriteLine(m.Value + " -> " + ans);
text = Post(url, "ques_ans=" + ans);
Console.WriteLine(text);
if (text.Contains("the_flag_is_")) break;
m = Regex.Match(text, #"\(.*\)");
}
}
}
But it does not work, because I always get back the 'Level 1' question. I used HttpWebRequest.CookieContainer property to reuse the cookies across different requests to keep up a session.
I don't know what is the problem. Maybe the session doesn't work.
Or perhaps HttpWebRequest is too slow to post back the result in time.
How to solve this automation task in C#?

Your code isn't handling being redirected. Watching the traffic we can see that this particular server would like your requests to /fastandfurious to end with a trailing slash. so change '../fastandfurious' to '../fastandfurious/' and that will fix it.

Python Requests Post Failing With 500 Error

I am using this excellent project on GitHub (https://github.com/cjyoung/MouseBitesWPF). However, that is written in C# and I really need something written in Python. I have boiled that code down to the absolute bare bones of what it needs to work and came up with this:
using System;
using System.Collections.Generic;
using System.Collections.ObjectModel;
using System.ComponentModel;
using System.IO;
using System.Linq;
using System.Net;
using System.Text;
using System.Text.RegularExpressions;
using System.Threading.Tasks;
namespace DisneyFinderSmaller
{
class Program
{
static CookieContainer cookieJar = new CookieContainer();
static internal string rootUrl = "https://disneyworld.disney.go.com";
static internal string siteUrl = "/dining/";
static internal string diningSearchUrl = "/finder/dining-availability";
static void Main(string[] args)
{
LaunchSearchInstance();
string test = Console.ReadLine();
}
private static void LaunchSearchInstance()
{
string result = getCookiesFromRequest(rootUrl + siteUrl, "", "GET");
string pep_csrf = "";
Match match = Regex.Match(result, "<input[^>]*name=['\"]pep_csrf['\"][^>]*value=['\"]([^'\"]*)['\"]*[^>]>", RegexOptions.Singleline & RegexOptions.IgnoreCase);
pep_csrf = match.Groups[1].ToString();
ConductSearch(pep_csrf);
}
private static void ConductSearch(string pep_csrf)
{
string postString = string.Format("&searchDate={1}" +
"&skipPricing=true" +
"&searchTime={2}" +
"&partySize={3}" +
"&id={0}%3BentityType%3Drestaurant" +
"&type=dining" +
"&pep_csrf={4}",
"293704",
"2015-11-18",
"80000714",
"2",
pep_csrf);
string result = getCookiesFromRequest(rootUrl + diningSearchUrl, postString, "POST");
System.Console.WriteLine(result);
}
private static String getCookiesFromRequest(string url, string postString, string method = "POST")
{
String result = "";
byte[] postBytes = Encoding.ASCII.GetBytes(postString);
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.Method = method;
request.Referer = rootUrl + siteUrl;
request.CookieContainer = cookieJar;
if (method == "POST")
{
request.ContentType = "application/x-www-form-urlencoded";
Stream postStream = request.GetRequestStream();
postStream.Write(postBytes, 0, postBytes.Length);
postStream.Close();
}
try
{
HttpWebResponse webResponse = (HttpWebResponse)request.GetResponse();
Stream responseStream = webResponse.GetResponseStream();
StreamReader responseStreamReader = new StreamReader(responseStream);
result = responseStreamReader.ReadToEnd();
responseStream.Close();
webResponse.Close();
}
catch (Exception ex)
{
Console.WriteLine("IOException source: {0}", ex.Source);
}
return result;
}
}
}
In my efforts to translate this to Python using Requests, I have come up with this:
#!/usr/bin/env python
import requests
url = "https://disneyworld.disney.go.com/dining/"
url2 = "https://disneyworld.disney.go.com/dining/finder/dining-availability"
session = requests.Session()
tokenRequest = session.get(url, headers=header)
start = tokenRequest.content.find('''id="pep_csrf"''')
pep = tokenRequest.content[start+21:tokenRequest.content.find('>',start+22)-1]
raw = "&searchDate=2015-11-18&skipPricing=true&searchTime=80000714&partySize=2&id=293704%3BentityType%3Drestaurant&type=dining&pep_csrf=" + pep
headers = {
'Content-Type': 'application/x-www-form-urlencoded',
'referer': 'https://disneyworld.disney.go.com/dining/',
'method' : 'POST'
}
result = session.post(url2, data=raw, headers=headers)
print result.status_code
But this doesn't work and returns a status code of 500.
Any thoughts on where things are going wrong? I have been hanging my head against the wall for a few days and any insight at all would be so appreciated.

calling google Url Shortner API in C#

I want to call the google url shortner API from my C# Console Application, the request I try to implement is:
POST https://www.googleapis.com/urlshortener/v1/url
Content-Type: application/json
{"longUrl": "http://www.google.com/"}
When I try to use this code:
using System.Net;
using System.Net.Http;
using System.IO;
and the main method is:
static void Main(string[] args)
{
string s = "http://www.google.com/";
var client = new HttpClient();
// Create the HttpContent for the form to be posted.
var requestContent = new FormUrlEncodedContent(new[] {new KeyValuePair<string, string>("longUrl", s),});
// Get the response.
HttpResponseMessage response = client.Post("https://www.googleapis.com/urlshortener/v1/url",requestContent);
// Get the response content.
HttpContent responseContent = response.Content;
// Get the stream of the content.
using (var reader = new StreamReader(responseContent.ContentReadStream))
{
// Write the output.
s = reader.ReadToEnd();
Console.WriteLine(s);
}
Console.Read();
}
I get the error code 400: This API does not support parsing form-encoded input.
I don't know how to fix this.

you can check the code below (made use of System.Net).
You should notice that the contenttype must be specfied, and must be "application/json"; and also the string to be send must be in json format.
using System;
using System.Net;
using System.IO;
namespace ConsoleApplication1
{
class Program
{
static void Main(string[] args)
{
var httpWebRequest = (HttpWebRequest)WebRequest.Create("https://www.googleapis.com/urlshortener/v1/url");
httpWebRequest.ContentType = "application/json";
httpWebRequest.Method = "POST";
using (var streamWriter = new StreamWriter(httpWebRequest.GetRequestStream()))
{
string json = "{\"longUrl\":\"http://www.google.com/\"}";
Console.WriteLine(json);
streamWriter.Write(json);
}
var httpResponse = (HttpWebResponse)httpWebRequest.GetResponse();
using (var streamReader = new StreamReader(httpResponse.GetResponseStream()))
{
var responseText = streamReader.ReadToEnd();
Console.WriteLine(responseText);
}
}
}
}

Google has a NuGet package for using the Urlshortener API. Details can be found here.
Based on this example you would implement it as such:
using System;
using System.Net;
using System.Net.Http;
using Google.Apis.Services;
using Google.Apis.Urlshortener.v1;
using Google.Apis.Urlshortener.v1.Data;
using Google.Apis.Http;
namespace ConsoleTestBed
{
class Program
{
private const string ApiKey = "YourAPIKey";
static void Main(string[] args)
{
var initializer = new BaseClientService.Initializer
{
ApiKey = ApiKey,
//HttpClientFactory = new ProxySupportedHttpClientFactory()
};
var service = new UrlshortenerService(initializer);
var longUrl = "http://wwww.google.com/";
var response = service.Url.Insert(new Url { LongUrl = longUrl }).Execute();
Console.WriteLine($"Short URL: {response.Id}");
Console.ReadKey();
}
}
}
If you are behind a firewall you may need to use a proxy. Below is an implementation of the ProxySupportedHttpClientFactory, which is commented out in the sample above. Credit for this goes to this blog post.
class ProxySupportedHttpClientFactory : HttpClientFactory
{
private static readonly Uri ProxyAddress
= new UriBuilder("http", "YourProxyIP", 80).Uri;
private static readonly NetworkCredential ProxyCredentials
= new NetworkCredential("user", "password", "domain");
protected override HttpMessageHandler CreateHandler(CreateHttpClientArgs args)
{
return new WebRequestHandler
{
UseProxy = true,
UseCookies = false,
Proxy = new WebProxy(ProxyAddress, false, null, ProxyCredentials)
};
}
}

How about changing
var requestContent = new FormUrlEncodedContent(new[]
{new KeyValuePair<string, string>("longUrl", s),});
to
var requestContent = new StringContent("{\"longUrl\": \" + s + \"}");

Following is my working code. May be its helpful for you.
private const string key = "xxxxxInsertGoogleAPIKeyHerexxxxxxxxxx";
public string urlShorter(string url)
{
string finalURL = "";
string post = "{\"longUrl\": \"" + url + "\"}";
string shortUrl = url;
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("https://www.googleapis.com/urlshortener/v1/url?key=" + key);
try
{
request.ServicePoint.Expect100Continue = false;
request.Method = "POST";
request.ContentLength = post.Length;
request.ContentType = "application/json";
request.Headers.Add("Cache-Control", "no-cache");
using (Stream requestStream = request.GetRequestStream())
{
byte[] postBuffer = Encoding.ASCII.GetBytes(post);
requestStream.Write(postBuffer, 0, postBuffer.Length);
}
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
using (Stream responseStream = response.GetResponseStream())
{
using (StreamReader responseReader = new StreamReader(responseStream))
{
string json = responseReader.ReadToEnd();
finalURL = Regex.Match(json, #"""id"": ?""(?.+)""").Groups["id"].Value;
}
}
}
}
catch (Exception ex)
{
// if Google's URL Shortener is down...
System.Diagnostics.Debug.WriteLine(ex.Message);
System.Diagnostics.Debug.WriteLine(ex.StackTrace);
}
return finalURL;
}

Uploading images to Imgur.com using C#

I just recieve my unique developer API key from Imgur and I'm aching to start cracking on this baby.
First a simple test to kick things off. How can I upload an image using C#? I found this using Python:
#!/usr/bin/python
import pycurl
c = pycurl.Curl()
values = [
("key", "YOUR_API_KEY"),
("image", (c.FORM_FILE, "file.png"))]
# OR: ("image", "http://example.com/example.jpg"))]
# OR: ("image", "BASE64_ENCODED_STRING"))]
c.setopt(c.URL, "http://imgur.com/api/upload.xml")
c.setopt(c.HTTPPOST, values)
c.perform()
c.close()

looks like the site uses HTTP Post to upload images. Take a look at the HTTPWebRequest class and using it to POST to a URL: Posting data with HTTPRequest.

The Imgur API now provide a complete c# example :
using System;
using System.IO;
using System.Net;
using System.Text;
namespace ImgurExample
{
class Program
{
static void Main(string[] args)
{
PostToImgur(#"C:\Users\ashwin\Desktop\image.jpg", IMGUR_ANONYMOUS_API_KEY);
}
public static void PostToImgur(string imagFilePath, string apiKey)
{
byte[] imageData;
FileStream fileStream = File.OpenRead(imagFilePath);
imageData = new byte[fileStream.Length];
fileStream.Read(imageData, 0, imageData.Length);
fileStream.Close();
string uploadRequestString = "image=" + Uri.EscapeDataString(System.Convert.ToBase64String(imageData)) + "&key=" + apiKey;
HttpWebRequest webRequest = (HttpWebRequest)WebRequest.Create("http://api.imgur.com/2/upload");
webRequest.Method = "POST";
webRequest.ContentType = "application/x-www-form-urlencoded";
webRequest.ServicePoint.Expect100Continue = false;
StreamWriter streamWriter = new StreamWriter(webRequest.GetRequestStream());
streamWriter.Write(uploadRequestString);
streamWriter.Close();
WebResponse response = webRequest.GetResponse();
Stream responseStream = response.GetResponseStream();
StreamReader responseReader = new StreamReader(responseStream);
string responseString = responseReader.ReadToEnd();
}
}
}

Why don't you use the NuGet for this: called Imgur.API and for upload
you would have a method like this:
/*
The refresh token and all the values represented by constans are given when you allow the application in your imgur panel on the response url
*/
public OAuth2Token CreateToken()
{
var token = new OAuth2Token(TOKEN_ACCESS, REFRESH_TOKEN, TOKEN_TYPE, ID_ACCOUNT, IMGUR_USER_ACCOUNT, int.Parse(EXPIRES_IN));
return token;
}
//Use it only if your token is expired
public Task<IOAuth2Token> RefreshToken()
{
var client = new ImgurClient(CLIENT_ID, CLIENT_SECRET);
var endpoint= new OAuth2Endpoint(client);
var token = endpoint.GetTokenByRefreshTokenAsync(REFRESH_TOKEN);
return token;
}
public async Task UploadImage()
{
try
{
var client = new ImgurClient(CLIENT_ID, CLIENT_SECRET, CreateToken());
var endpoint = new ImageEndpoint(client);
IImage image;
//Here you have to link your image location
using (var fs = new FileStream(#"IMAGE_LOCATION", FileMode.Open))
{
image = await endpoint.UploadImageStreamAsync(fs);
}
Debug.Write("Image uploaded. Image Url: " + image.Link);
}
catch (ImgurException imgurEx)
{
Debug.Write("Error uploading the image to Imgur");
Debug.Write(imgurEx.Message);
}
}
Also you can find all the reference here: Imgur.API NuGet