We Keep Coding

How to set auth cookie for WebClient.downloadstring behind login form

I am trying to use WebClient.DownloadString() to scrape JSON data from a URL.
The issue is I find that programmatically accessing the URL: "secure.somesite.com.au/api/products/getprice?productName=Cornmeal"
results in the site forcefully closing the connection.
I believe this occurs because the auth cookie is not set.
How does one set a cookie? I've spent some time reading up on stackoverflow and codeproject, and no one is setting actual cookies, they're all setting username and passwords. I need to set the cookie so the site knows I should have access.
using (var client = new CookieAwareWebClient())
{
Cookie cookie = new Cookie();
cookie.Name = "SWI";
cookie.Value = "kjuujj7kxPvEC-4fBt5yyzWOJnjhriuoOtZ6Z0Ww";
cookie.Domain = ".secure.somesite.com.au";
client.CookieContainer.Add(cookie);
string r = client.DownloadString("https://secure.somesite.com.au/api/products/getprice?productName=Cornmeal");
}
CookieAwareWebClient Class:
public class CookieAwareWebClient : WebClient
{
public CookieAwareWebClient()
{
CookieContainer = new CookieContainer();
}
public CookieContainer CookieContainer { get; private set; }
protected override WebRequest GetWebRequest(Uri address)
{
var request = (HttpWebRequest)base.GetWebRequest(address);
request.CookieContainer = CookieContainer;
return request;
}
}
PS. I have attempted to login with WebClient and all I get is the connection is closed forcefully. I think this is because if you are not logged already, requesting protected resources results in an error in WebClient and not just a string saying "null" being returned or something.
PPS. I have done this in python, but now need it working in C#.
client.get('https://secure.somesite.com.au/api/products/getprice', params={
'productCode': '{}'.format(code)
}, headers=headers, timeout=60)
The cookie here is in a header.

c# WebClient catch Response Cookies and collect all of cookies for make a new successful requests

I am trying to login a website then walk around for catch some infromation and do some stuff in it.
Everything is well when the request doesnt need any cookies, but some page need cookie which is created on first request. So I should collect all of cookies in WebClient object.
I am using this code but its not enough for me. Because still I am missing all of my cookies for next request.
public class CookieAwareWebClient : WebClient
{
public CookieContainer CookieContainer { get; set; }
public CookieCollection ResponseCookies { get; set; }
//public CookieContainer ResponseCookieContainer { get; set; }
public CookieAwareWebClient()
: base()
{
CookieContainer = new CookieContainer();
ResponseCookies = new CookieCollection();
}
protected override WebRequest GetWebRequest(Uri address)
{
WebRequest request = base.GetWebRequest(address);
HttpWebRequest webRequest = request as HttpWebRequest;
if (webRequest != null)
{
webRequest.CookieContainer = CookieContainer;
}
return request;
}
protected override WebResponse GetWebResponse(WebRequest request)
{
var response = (HttpWebResponse)base.GetWebResponse(request);
this.ResponseCookies = response.Cookies;
return response;
}
Here is my code for make request.
var loginLink ="https...."; // an Uri with username and password values as queryString
CookieAwareWebClient client = new CookieAwareWebClient();
var loginResult = client.DownloadString(loginLink);
I can see the result , yes I am in! login was success, but I am losting all of my cookies , and next request sending me to login page..
I should collect all of cookies in my webcilent cookieContainer. I should read Response Header for set "Set-Cookie" values to my container.
Think about this, I have 3 key in my cookies before request.
a = "123",
b = "asd",
c = "123"
and now I send a request to website and its returning me 2 cookies(one is new and one is old key with new value) back,(I can see in response Headers , in "Set-Cookie")
a = "123456",
d = "blabla"
so I need to change key of "a"s value, and I need to add "d" key to my Cookie, because I dont want to go back to login page on my next request again...
Maybe I need a library, maybe a better WebClient Class whcih can collect all cokkies to help me to discover all pages.
I hope someone can help me.
Best Regards!

c# Script to login and download from Kaggle

Recently, I came across a python script to download files directly from Kaggle : https://ramhiser.com/2012/11/23/how-to-download-kaggle-data-with-python-and-requests-dot-py/
I am trying to do something similar using WebClients in C#. I've came the following response in StackOverFlow : C# download file from the web with login
Tried using it but I seem to be downloading only the login page instead of the actual file. Here's my main code :
CookieContainer cookieJar = new CookieContainer();
CookieAwareWebClient http = new CookieAwareWebClient(cookieJar);
string postData = "name=<username>&password=<password>&submit=submit";
string response = http.UploadString("https://www.kaggle.com/account/login", postData);
Console.Write(response);
http.DownloadFile("https://www.kaggle.com/c/titanic/download/train.csv", "train.CSV");
I've used the Webclient extension from the link above and modified slightly :
public class CookieAwareWebClient : WebClient
{
public CookieContainer CookieContainer { get; set; }
public Uri Uri { get; set; }
public CookieAwareWebClient()
: this(new CookieContainer())
{
}
public CookieAwareWebClient(CookieContainer cookies)
{
this.CookieContainer = cookies;
}
protected override WebRequest GetWebRequest(Uri address)
{
this.Uri = address;
WebRequest request = base.GetWebRequest(address);
if (request is HttpWebRequest)
{
(request as HttpWebRequest).CookieContainer = this.CookieContainer;
}
HttpWebRequest httpRequest = (HttpWebRequest)request;
httpRequest.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
return httpRequest;
}
protected override WebResponse GetWebResponse(WebRequest request)
{
WebResponse r = base.GetWebResponse(request);
var response = r as HttpWebResponse;
if (response != null)
{
CookieCollection cookies = response.Cookies;
CookieContainer.Add(cookies);
}
return response;
}
}
Was wondering if anyone can point out where I went wrong?
Thanks.

We have created a forum post to help you accomplish what you wanted to do, Accessing Kaggle API through C#. Feel free to post here or on the forum if you have additional questions.

Try to go to https://www.kaggle.com/c/titanic/download/train.csv by your browser without logged in and your browser will open that page instead of downloading your file. You need to put direct link to the file instead of a web page.
Your code works perfectly, you just need to put a direct link to that file or make sure you have logged in before download the file.

I know it's not exactly what you were asking, but Kaggle now has an official API that you can use to download data. Should be a bit easier to use. :)

WebClient Does not automatically redirect

When logging the login process using Firebug i see that it is like this
POST //The normal post request
GET //Automatically made after the login
GET //Automatically made after the login
GET //Automatically made after the login
When making a post request using my code below it did not make the automatic GET requests that the browsers is doing.
MY WebClient Handler
using System;
using System.Net;
namespace Test
{
class HttpHandler : WebClient
{
private CookieContainer _mContainer = new CookieContainer();
protected override WebRequest GetWebRequest(Uri address)
{
var request = base.GetWebRequest(address);
if (request is HttpWebRequest)
{
(request as HttpWebRequest).CookieContainer = _mContainer;
}
return request;
}
protected override WebResponse GetWebResponse(WebRequest request)
{
var response = base.GetWebResponse(request);
if (response is HttpWebResponse)
_mContainer.Add((response as HttpWebResponse).Cookies);
return response;
}
public void ClearCookies()
{
_mContainer = new CookieContainer();
}
}
}
Using Code
private static async Task<byte[]> LoginAsync(string username, string password)
{
var postData = new NameValueCollection();
var uri = new Uri(string.Format("http://{0}/", ServerName));
postData.Add("name", username);
postData.Add("password", password);
return await HttpHandler.UploadValuesTaskAsync(uri, postData);
}
When trying to track the connection of my application it is only doing the POST Request and not the rest of GET requests. [THAT ARE MADE AUTOMATICALLY IN THE BROWSER]

Try adding
request.AllowAutoRedirect = true;
right under the
var request = base.GetWebRequest(address);
It solved some similar problems for me, even though AllowAutoRedirect is supposed to be true by default.

That shouldn't be surprising, given that HttpWebRequest is not a browser. If you need to perform these redirects, then check the HttpWebResponse.StatusCode, and make another request if it's a redirect code in the 300's. Note from the link under 10.3 Redirection 3xx:
This class of status code indicates that further action needs to be taken by the user agent in order to fulfill the request. The action required MAY be carried out by the user agent without interaction with the user if and only if the method used in the second request is GET or HEAD. A client SHOULD detect infinite redirection loops, since such loops generate network traffic for each redirection.

How to Send a Web Request to Log Out?

I want to log out from page using webclient.
This is my code for login and site downloading.
public bool LogIn(string loginName, string password)
{
try
{
NameValueCollection postData = new NameValueCollection();
postData.Add("login", loginName);
postData.Add("password", password);
// Authenticate
_webClient.UploadValues("http://rapideo.pl/login.php", postData);
//string temp = _webClient.DownloadString("http://rapideo.pl/lista");
}
catch
{
return false;
}
_loggedIn = true;
_loginName = loginName;
return true;
}
class WebClientEx : WebClient
{
public CookieContainer CookieContainer { get; private set; }
public WebClientEx()
{
CookieContainer = new CookieContainer();
}
protected override WebRequest GetWebRequest(Uri address)
{
var request = base.GetWebRequest(address);
if (request is HttpWebRequest)
{
(request as HttpWebRequest).CookieContainer = CookieContainer;
}
return request;
}
}
In order to logout I only need to open that page in browser:
http://rapideo.pl/wyloguj
I know how to download sourcecode of the page after login.
But how can I send http request to logout? I don't want do get response or sourcecode of that page. i just want to sent request.

As a sanity check, have you already tried doing a WebRequest.DownloadString("http://rapideo.pl/wyloguj") and then just discarding the returned data?
If that is not working, one thing to try would be to look at the request/response messages in a tool like Fiddler to see what exactly is going over the wire when you log out via the browser versus programmatically.
Also, as a general aside, it looks like the user's name and password are being sent in the clear as part of the login. Not sure if there is an HTTPS login endpoint available for that site but that would be something to look into.