We Keep Coding

HttpWebReponse Cookies Not Setting For Redirect

I'm trying to do the following:
send a GET request to fetch a login page (which prompts for username, password, and sets a cookie)
build a POST request that sends the cookie from #1 and a body of the username/password (this returns a Set-Cookie and redirects to the website's landing page for logged in users)
My trouble is with the 302 redirect. The webserver is returning a 302 with a Set-Cookie, but when the HttpWebRequests auto-redirects, it doesn't pass along the now updated cookie. To get around that, I'm trying to set .AllowAutoRedirect = false, saving the cookies in a CookieCollection, then building a 3rd HTTP request: a GET to the final 302 location. Unfortunately, I can't set the cookies on this request. I'm not sure why and it's driving me mad.
The HTTP requests are, in order, named request, postRequest, redirectRequest.
string loginGetUrl = "https://<..>/signin.htm";
string loginPostUrl = "https://<..>/j_acegi_security_check";
string loginRedirectUrl = "https://<..>/centraladmin/poslinks.htm";
string postData = String.Format("j_username={0}&j_password={1}", username, password);
CookieCollection cookies = new CookieCollection();
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(loginGetUrl);
request.CookieContainer = new CookieContainer();
request.CookieContainer.Add(cookies);
//Get the response from the server and save the cookies from the first request..
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
cookies = response.Cookies;
HttpWebRequest postRequest = (HttpWebRequest)WebRequest.Create(loginPostUrl);
postRequest.CookieContainer = new CookieContainer();
// Add the received Cookies from the HTTP Get
postRequest.CookieContainer.Add(cookies);
postRequest.Method = WebRequestMethods.Http.Post;
postRequest.UserAgent = "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.121 Safari/535.2";
postRequest.AllowWriteStreamBuffering = false;
postRequest.ProtocolVersion = HttpVersion.Version11;
postRequest.AllowAutoRedirect = false;
postRequest.ContentType = "application/x-www-form-urlencoded";
byte[] byteArray = Encoding.ASCII.GetBytes(postData);
postRequest.ContentLength = byteArray.Length;
Stream newStream = postRequest.GetRequestStream(); //open connection
newStream.Write(byteArray, 0, byteArray.Length); // Send the data.
newStream.Close();
HttpWebResponse postResponse = (HttpWebResponse)postRequest.GetResponse();
// Save the cookies from the POST login request, then send them on to the redirected URL
cookies = postResponse.Cookies;
HttpWebRequest redirectRequest = (HttpWebRequest)WebRequest.Create(loginRedirectUrl);
redirectRequest.CookieContainer = new CookieContainer();
// add cookies from POST
redirectRequest.CookieContainer.Add(cookies);
HttpWebResponse redirectResponse = (HttpWebResponse)redirectRequest.GetResponse();
At redirectRequest.CookieContainer.Add(cookies);, the cookies object contains the correct cookie. But when I look with Fiddler, I only see this info:
GET https://<...>/centraladmin/poslinks.htm HTTP/1.1
Host: host:port
I'm kind of banging my head on the wall at this point. Any suggestions? Am I referencing something wrong? Beware, I don't usually write C# code

I wasn't able to resolve this on my own, but did find a useful code snippet from this blog post by #malte-clasen. The code is on Github and I've attached it here for retention.
I removed the async components as it wasn't necessary in my code.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Net;
using System.Text;
using System.Threading.Tasks;
namespace DracWake.Core
{
public class WebClient : IWebClient
{
private readonly CookieContainer _cookies = new CookieContainer();
private HttpWebRequest CreateRequest(Uri uri)
{
var request = HttpWebRequest.CreateHttp(uri);
request.AllowAutoRedirect = false;
request.CookieContainer = _cookies;
SetHeaders(request);
var defaultValidator = System.Net.ServicePointManager.ServerCertificateValidationCallback;
request.ServerCertificateValidationCallback =
(sender, certificate, chain, sslPolicyErrors) =>
certificate.Subject.Contains("O=DO_NOT_TRUST, OU=Created by http://www.fiddler2.com")
|| (certificate.Subject == "CN=DRAC5 default certificate, OU=Remote Access Group, O=Dell Inc., L=Round Rock, S=Texas, C=US")
|| (defaultValidator != null && defaultValidator(request, certificate, chain, sslPolicyErrors));
return request;
}
private async Task<string> DecodeResponse(HttpWebResponse response)
{
foreach (System.Net.Cookie cookie in response.Cookies)
{
_cookies.Add(new Uri(response.ResponseUri.GetLeftPart(UriPartial.Authority)), cookie);
}
if (response.StatusCode == HttpStatusCode.Redirect)
{
var location = response.Headers[HttpResponseHeader.Location];
if (!string.IsNullOrEmpty(location))
return await Get(new Uri(location));
}
var stream = response.GetResponseStream();
var buffer = new System.IO.MemoryStream();
var block = new byte[65536];
var blockLength = 0;
do{
blockLength = stream.Read(block, 0, block.Length);
buffer.Write(block, 0, blockLength);
}
while(blockLength == block.Length);
return Encoding.UTF8.GetString(buffer.GetBuffer());
}
public async Task<string> Get(Uri uri)
{
var request = CreateRequest(uri);
var response = (HttpWebResponse) await request.GetResponseAsync();
return await DecodeResponse(response);
}
private void SetHeaders(HttpWebRequest request)
{
request.Accept = "text/html, application/xhtml+xml, */*";
request.UserAgent = "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; WOW64; Trident/6.0)";
request.ContentType = "application/x-www-form-urlencoded";
request.Headers[HttpRequestHeader.AcceptLanguage] = "en-US,en;q=0.8,de-DE;q=0.5,de;q=0.3";
request.Headers[HttpRequestHeader.AcceptEncoding] = "gzip, deflate";
request.Headers[HttpRequestHeader.CacheControl] = "no-cache";
}
public async Task<string> Post(Uri uri, byte[] data)
{
var request = CreateRequest(uri);
request.Method = "POST";
request.GetRequestStream().Write(data, 0, data.Length);
var response = (HttpWebResponse) await request.GetResponseAsync();
return await DecodeResponse(response);
}
}
}
The DecodeResponse resolved my problem.

c# web browser cookies in httpwebrequest

I have a webbrowser control which loads a page.
I then hit a button to call this method:
public void get(Uri myUri)
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(myUri);
CookieContainer cookieJar = new CookieContainer();
cookieJar.SetCookies(webBrowser1.Document.Url,webBrowser1.Document.Cookie.Replace(';', ','));
request.CookieContainer = cookieJar;
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
int cookieCount = cookieJar.Count;
Stream receiveStream = response.GetResponseStream();
StreamReader readStream = new StreamReader(receiveStream, Encoding.UTF8);
txt.Text = readStream.ReadToEnd();
txt2.Text = cookieCount.ToString();
}
As from the cookieCount int i can see that if i call the method before logging in on the page in the web browser control i would get 6 cookies, and after i log in i get 7.
However, even with the cookies the response i get is the same as if i wouldnt have been logged in.
So i am guessing that the cookies isnt being sent with the request?
Thanks!

You're recreating your CookieContainer every time you call this method, you need to use the same CookieContainer in all requests
you can use this code, to handle your requests:
static CookieContainer cookies = new CookieContainer();
static HttpWebRequest GetNewRequest(string targetUrl, CookieContainer SessionCookieContainer)
{
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(targetUrl);
request.CookieContainer = SessionCookieContainer;
request.AllowAutoRedirect = false;
return request;
}
public static HttpWebResponse MakeRequest(HttpWebRequest request, CookieContainer SessionCookieContainer, Dictionary<string, string> parameters = null)
{
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.52 Safari/536.5Accept: */*";
request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
request.CookieContainer = SessionCookieContainer;
request.AllowAutoRedirect = false;
if (parameters != null)
{
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded";
string postData = "";
foreach (KeyValuePair<String, String> parametro in parameters)
{
if (postData.Length == 0)
{
postData += String.Format("{0}={1}", parametro.Key, parametro.Value);
}
else
{
postData += String.Format("&{0}={1}", parametro.Key, parametro.Value);
}
}
byte[] postBuffer = UTF8Encoding.UTF8.GetBytes(postData);
using (Stream postStream = request.GetRequestStream())
{
postStream.Write(postBuffer, 0, postBuffer.Length);
}
}
else
{
request.Method = "GET";
}
HttpWebResponse response = request.GetResponse() as HttpWebResponse;
SessionCookieContainer.Add(response.Cookies);
while (response.StatusCode == HttpStatusCode.Found)
{
response.Close();
request = GetNewRequest(response.Headers["Location"], SessionCookieContainer);
response = (HttpWebResponse)request.GetResponse();
SessionCookieContainer.Add(response.Cookies);
}
return response;
}
and to request a page,
HttpWebRequest request = GetNewRequest("http://www.elitepvpers.com/forum/login.php?do=login", cookies);
Dictionary<string,string> parameters = new Dictionary<string,string>{{"your params","as key value"};
HttpWebResponse response = MakeRequest(request, cookies, parameters);
using (StreamReader reader = new StreamReader(response.GetResponseStream()))
{
if(!reader.EndOfStream)
{
Console.Write(reader.ReadToEnd());
}
}

When matching the session, your web server may be taking into account some other HTTP request headers, besides cookies. To name a few: User-Agent, Authorization, Accept-Language.
Because WebBrowser control and WebRequest do not share sessions, you'd need to replicate all headers from the WebBrowser session. This would be hard thing to do, as you'd need to intercept WebBrowser trafic, in a way similar to what Fiddler does.
A more feasible solution might be to stay on the same session with WebBrowser by using Windows UrlMon APIs like URLOpenStream, URLDownloadToFile etc., instead of WebRequest. That works, because WebBrowser uses UrlMon library behind the scene.
I've recently answered some related questions:
The notorious yet unaswered issue of downloading a file when windows security is required
Upload a file to a website programmatically?

Login to Https Website with WebRequest and WebResponse Not Working C#

I am working on a program that will log into a website and get certian data. However I am having trouble posting the login parameters and dealing with the cookies, as each time I get a page saying "You have logged out or Session has expired." So clearly I'm doing something wrong with posting the parameters or dealing with the cookies, but don't know which. I have been working on this for a while and just can't get my head around why this is not working correctly.
void Login2(string username, string password)
{
string pageSource;
string formUrl = "https://forUrl.com";
string formParams = string.Format("login={0}&sslProt={1}&pwd={2}&gru={3}", username, "", password, "115237091");
string cookieHeader;
HttpWebRequest req = (HttpWebRequest)WebRequest.Create(formUrl);
req.AllowAutoRedirect = false;
req.ContentType = "application/x-www-form-urlencoded";
req.Method = "POST";
byte[] bytes = System.Text.Encoding.ASCII.GetBytes(formParams);
req.ContentLength = bytes.Length;
using (Stream os = req.GetRequestStream())
{
os.Write(bytes, 0, bytes.Length);
}
HttpWebResponse resp = (HttpWebResponse)req.GetResponse();
cookieHeader = resp.Headers["Set-cookie"];
string getUrl = "https://Urlbehindform.com";
HttpWebRequest getRequest = (HttpWebRequest)WebRequest.Create(getUrl);
getRequest.Method = "GET";
getRequest.AllowAutoRedirect = false;
getRequest.Headers.Add("Cookie", cookieHeader);
HttpWebResponse getResponse = (HttpWebResponse)getRequest.GetResponse();
using (StreamReader sr = new StreamReader(getResponse.GetResponseStream()))
{
pageSource = sr.ReadToEnd();
}
Response.Redirect(getUrl);
}
I am getting the cookie when I do the POST and sending it back when I do the GET, but for some reason this doesn't seem to work. At first I thought it was the parameters, but after looking at the issue further using Tamper Data with Firefox the login parameters seem to be working fine. Any help would be great, as I have been working on this for a while and can't wrap my head around it. Thanks!
UPDATE:
After trying out a few suggestions I still can't get this to work. However Upon looking deeper into Data Tamper, It appears that there is a POST with the login parameters, then a GET to a different page and then finally the GET to the page after the login page (The one I'm trying to get to). After some further debugging I actually discovered that my login POST is not working as I thought, As the response header location is showing "/cv/scripts/A028/eng/logErr.asp". Meaning the rest of my code could have been fine all a long, it was that the POST wasn't giving me a valid login. Any Sugguestions as to why I am always getting the login error page? As always thanks for the help.
UPDATE:
After playing around further with Tamper Data is appears that the reason I am unable to get a successful login is that in order have a successful POST of the parameters there needs to be a cookie already obtained. How do I go about doing this?

Use a single CookieContainer for both requests. Then you don't have to copy cookies manually.
I [BMW1] added in a CookieContainer called cookies, but it still not working, Im not sure if im using the CookieContainer the right way. Here is an updated version of my code.
And edited by me [Hans Kesting], see comments with [HK]
void Login2(string username, string password)
{
string pageSource;
string formUrl = "https://server/cv/scripts/A028/eng/logProc.asp?ntry=0&dbg=";
string formParams = string.Format("login={0}&sslProt={1}&pwd={2}&gru={3}", username, "", password, "115237091");
// [HK] create a container for the cookies, where they are added automatically
CookieContainer cookies = new CookieContainer();
HttpWebRequest req = (HttpWebRequest)WebRequest.Create(formUrl);
req.CookieContainer = cookies;
req.AllowAutoRedirect = false;
req.ContentType = "application/x-www-form-urlencoded";
req.Method = "POST";
byte[] bytes = System.Text.Encoding.ASCII.GetBytes(formParams);
req.ContentLength = bytes.Length;
using (Stream os = req.GetRequestStream())
{
os.Write(bytes, 0, bytes.Length);
}
HttpWebResponse resp = (HttpWebResponse)req.GetResponse();
// [HK] no need to add cookies "by hand", that will happen automatically
//cookies.Add(resp.Cookies);
string getUrl = "https://server/cv/scripts/A028/eng/home.asp";
HttpWebRequest getRequest = (HttpWebRequest)WebRequest.Create(getUrl);
// [HK] use the same cookiecontainer as on the first request - correct
getRequest.CookieContainer = cookies;
getRequest.Method = "GET";
getRequest.AllowAutoRedirect = false;
HttpWebResponse getResponse = (HttpWebResponse)getRequest.GetResponse();
// [HK] no need to add cookies, they should be there already
//cookies.Add(getResponse.Cookies);
using (StreamReader sr = new StreamReader(getResponse.GetResponseStream()))
{
pageSource = sr.ReadToEnd();
}
// [HK] no need to add cookies, they should be there already
// cookies.Add(getResponse.Cookies);
Response.Redirect(getUrl);
}

You could use a Cookie aware web client,
public class CookieAwareWebClient : WebClient
{
public CookieContainer CookieContainer { get; set; }
public Uri Uri { get; set; }
public CookieAwareWebClient() : this (new CookieContainer())
{
}
public CookieAwareWebClient(CookieContainer cookies)
{
this.CookieContainer = cookies;
}
protected override WebRequest GetWebRequest(Uri address)
{
WebRequest request = base.GetWebRequest(address);
if (request is HttpWebRequest)
{
(request as HttpWebRequest).CookieContainer = this.CookieContainer;
}
HttpWebRequest httpRequest = (HttpWebRequest) request;
httpRequest.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
return httpRequest;
}
protected override WebResponse GetWebResponse(WebRequest request)
{
WebResponse response = base.GetWebResponse(request);
String setCookieHeader = response.Headers[HttpResponseHeader.SetCookie];
if (setCookieHeader != null)
{
//do something if needed to parse out the cookie.
if (setCookieHeader != null)
{
Cookie cookie = new Cookie(); //create cookie
this.CookieContainer.Add(cookie);
}
}
return response;
}
}
Example usage:
var wc = new CookieAwareWebClient ();
wc.Headers["Content-type"] = "application/x-www-form-urlencoded";
string HtmlResult = wc.UploadString(URI, myParameters);

c# Post data cookies not working

Alright, so I've got a function I wrote that should allow me to post data with cookies. The problem is is I'm testing it out on an Amazon login page, and it keeps responding saying I need cookies enabled. Here's the code
public string DoPost(String url, PostData data, CookieContainer cookies)
{
HttpWebRequest objWebRequest = (HttpWebRequest)WebRequest.Create(url);
objWebRequest.CookieContainer = cookies;
objWebRequest.AllowAutoRedirect = true;
objWebRequest.UserAgent = "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)";
if(data != null)
{
String post = data.ToString();
objWebRequest.Method = "POST";
objWebRequest.ContentLength = post.Length;
objWebRequest.ContentType = "application/x-www-form-urlencoded";
// Post to the login form.
using(StreamWriter swRequestWriter = new StreamWriter(objWebRequest.GetRequestStream()))
{
swRequestWriter.Write(post);
}
}
// Get the response.
HttpWebResponse objWebResponse =
(HttpWebResponse)objWebRequest.GetResponse();
// Read the response
using(StreamReader srResponseReader = new StreamReader(objWebResponse.GetResponseStream()))
{
string strResponseData = srResponseReader.ReadToEnd();
return strResponseData;
}
}
And I call it like this
String action = "https://www.amazon.com/gp/flex/sign-in/select.html";
String s = DoPost(action, null, Cookies);
Cookies is created in my class constructer like this
CookieContainer Cookies;
public Constructz0r()
{
Cookies = new CookieContainer();
}
The thing is, I'm not even posting any post data, I'm just going to the page, and it's saying my cookies aren't enabled, though I feel I've done it write in DoPost.
I've even tried using this implementation of WebClient
public class CookieWebClient : WebClient
{
private CookieContainer _cookieContainer = new CookieContainer();
protected override WebRequest GetWebRequest(Uri address)
{
WebRequest request = base.GetWebRequest(address);
if (request is HttpWebRequest)
{
(request as HttpWebRequest).CookieContainer = _cookieContainer;
}
return request;
}
}
And calling it like this
using(CookieWebClient ck = new CookieWebClient())
{
String s = ck.DownloadString(action);
}
And it still tells me the cookies aren't enabled.

Amazon have an API to access their services (SOAP). So instead of trying to do some scraping I would strongly recommend you using their API.

HttpWebRequest pass credentials to next HttpWebRequest

I am logging into a page using HttpWebRequest and getting some information. I then use that information to create a new HttpWebRequest to get some more information. I do not want to use WebClient.
How can I pass the credentials I obtained from logging in using the first HttpWebRequest to the second one?
EDIT: If I use a CookieCollection then this is coming back as empty. I just tried using WebClient as a last resort and even for that it is not working, the second request takes me back to the login screen. I noticed that in a WebBrowser there is a cookie.

Add a CookieContainer to each request before you send it. Add the cookies you get from the first response to the second request. Assuming they use cookies for authentication, this should authenticate the second request.
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(urlWithParameters);
request.CookieContainer = new CookieContainer();
HttpWebResponse response = (HttpWebResponse) request.GetResponse();
var cookies = new CookieContainer();
cookies.Add( response.Cookies );
request = (HttpWebRequest)WebRequest.Create(secondUrlWithParameters);
request.CookieContainer = cookies;
...

this is just a sample running code based on answer 2. Maybe be redundant maybe illustrate somebody.
string url = "http://servername/place-where-data-is.extension"
string loginUrl = "https://servername/sampleLogin?email=eeeeee&passwd=xxxxxxx";
HttpWebRequest loginRequest = (HttpWebRequest)HttpWebRequest.Create(loginUrl);
loginRequest.CookieContainer = new CookieContainer();
loginRequest.Method = WebRequestMethods.Http.Get;
HttpWebResponse loginResponse = (HttpWebResponse)loginRequest.GetResponse();
var cookies = new CookieContainer();
cookies.Add(loginResponse.Cookies);
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(url);
request.CookieContainer = cookies;
request.Method = WebRequestMethods.Http.Get;
WebResponse response = (WebResponse)request.GetResponse();
Stream responseStream = response.GetResponseStream();

This is a really old question and I know it states no WebClient but I will post here for everyone who comes across this from Google. The original concept is not my code. I do not know where I originally found it.
using (WebClientEx client = new WebClientEx())
{
client.IntTimeout = intTimeout;
client.DownloadString(strReportUrlPrefix + strReportUrlQuery);
NameValueCollection auth = new NameValueCollection
{
{ "j_username", strReportUsername},
{ "j_password", strReportPassword}
};
byte[] data = client.UploadValues(strReportUrlPrefix + "j_security_check", auth);
// LOGIC HERE WITH DATA
}
WebClientEx Class:
public class WebClientEx : WebClient
{
private CookieContainer _cookieContainer = new CookieContainer();
public int IntTimeout { get; set; }
protected override WebRequest GetWebRequest(Uri address)
{
WebRequest request = base.GetWebRequest(address);
if (request != null)
request.Timeout = IntTimeout;
if (request is HttpWebRequest)
(request as HttpWebRequest).CookieContainer = _cookieContainer;
return request;
}
}