I am trying to unit test some code, and I need to to replace this:
HttpWebRequest httpWebRequest = (HttpWebRequest)WebRequest.Create( uri );
httpWebRequest.CookieContainer = new CookieContainer();
with
WebRequest webRequest = WebRequest.Create( uri );
webRequest.CookieContainer = new CookieContainer();
Basically, how do I get cookies into the request without using a HttpWebRequest?
Based on your comments, you might consider writing an extension method:
public static bool TryAddCookie(this WebRequest webRequest, Cookie cookie)
{
HttpWebRequest httpRequest = webRequest as HttpWebRequest;
if (httpRequest == null)
{
return false;
}
if (httpRequest.CookieContainer == null)
{
httpRequest.CookieContainer = new CookieContainer();
}
httpRequest.CookieContainer.Add(cookie);
return true;
}
Then you can have code like:
WebRequest webRequest = WebRequest.Create( uri );
webRequest.TryAddCookie(new Cookie("someName","someValue"));
Try with something like this:
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://www.contoso.com/default.html");
request.CookieContainer = new CookieContainer();
request.CookieContainer.Add(new Cookie("ConstoCookie", "Chocolate Flavour"));
WebRequest is an abstract class that does not have a CookieContainer property. In addition you can't use the Headers collection (not implemented exception) so any attempt like webRequest.Headers.Add("Cookie", "...") will fail.
Sorry, but you have no chance to use cookies with WebRequest.
Stick on HttpWebRequest and add/edit as many cookies you like using its Headers collection!
dlev's answer ended up working, but I had problems implementing the solution ("The parameter '{0}' cannot be an empty string."), so I decided to write the full code in case anybody else has similar problems.
My goal was to get the html as a string, but I needed to add the cookies to the web request. This is the function that downloads the string using the cookies:
public static string DownloadString(string url, Encoding encoding, IDictionary<string, string> cookieNameValues)
{
using (var webClient = new WebClient())
{
var uri = new Uri(url);
var webRequest = WebRequest.Create(uri);
foreach(var nameValue in cookieNameValues)
{
webRequest.TryAddCookie(new Cookie(nameValue.Key, nameValue.Value, "/", uri.Host));
}
var response = webRequest.GetResponse();
var receiveStream = response.GetResponseStream();
var readStream = new StreamReader(receiveStream, encoding);
var htmlCode = readStream.ReadToEnd();
return htmlCode;
}
}
We are using the code from dlev's answer:
public static bool TryAddCookie(this WebRequest webRequest, Cookie cookie)
{
HttpWebRequest httpRequest = webRequest as HttpWebRequest;
if (httpRequest == null)
{
return false;
}
if (httpRequest.CookieContainer == null)
{
httpRequest.CookieContainer = new CookieContainer();
}
httpRequest.CookieContainer.Add(cookie);
return true;
}
This is how you use the full code:
var cookieNameValues = new Dictionary<string, string>();
cookieNameValues.Add("varName", "varValue");
var htmlResult = DownloadString(url, Encoding.UTF8, cookieNameValues);
Related
I am making a call to a page on my site using webclient. I'm trying to get the result of the webpage put into a pdf so I am trying to get a string representation of the rendered page. The problem is that the request is not authenticated so all I get is a login screen. I have sent the UseDefaultCredentials property to true but I still get the same result. Below is a portion of my code:
WebClient webClient = new WebClient();
webClient.Encoding = Encoding.UTF8;
webClient.UseDefaultCredentials = true;
return Encoding.UTF8.GetString(webClient.UploadValues(link, "POST",form));
You need to give the WebClient object the credentials. Something like this...
WebClient client = new WebClient();
client.Credentials = new NetworkCredential("username", "password");
What kind of authentication are you using? If it's Forms authentication, then at best, you'll have to find the .ASPXAUTH cookie and pass it in the WebClient request.
At worst, it won't work.
Public Function getWeb(ByRef sURL As String) As String
Dim myWebClient As New System.Net.WebClient()
Try
Dim myCredentialCache As New System.Net.CredentialCache()
Dim myURI As New Uri(sURL)
myCredentialCache.Add(myURI, "ntlm", System.Net.CredentialCache.DefaultNetworkCredentials)
myWebClient.Encoding = System.Text.Encoding.UTF8
myWebClient.Credentials = myCredentialCache
Return myWebClient.DownloadString(myURI)
Catch ex As Exception
Return "Exception " & ex.ToString()
End Try
End Function
This helped me to call API that was using cookie authentication. I have passed authorization in header like this:
request.Headers.Set("Authorization", Utility.Helper.ReadCookie("AuthCookie"));
complete code:
// utility method to read the cookie value:
public static string ReadCookie(string cookieName)
{
var cookies = HttpContext.Current.Request.Cookies;
var cookie = cookies.Get(cookieName);
if (cookie != null)
return cookie.Value;
return null;
}
// using statements where you are creating your webclient
using System.Web.Script.Serialization;
using System.Net;
using System.IO;
// WebClient:
var requestUrl = "<API_url>";
var postRequest = new ClassRoom { name = "kushal seth" };
using (var webClient = new WebClient()) {
JavaScriptSerializer serializer = new JavaScriptSerializer();
byte[] requestData = Encoding.ASCII.GetBytes(serializer.Serialize(postRequest));
HttpWebRequest request = WebRequest.Create(requestUrl) as HttpWebRequest;
request.Method = "POST";
request.ContentType = "application/json";
request.ContentLength = requestData.Length;
request.ContentType = "application/json";
request.Expect = "application/json";
request.Headers.Set("Authorization", Utility.Helper.ReadCookie("AuthCookie"));
request.GetRequestStream().Write(requestData, 0, requestData.Length);
using (var response = (HttpWebResponse)request.GetResponse()) {
var reader = new StreamReader(response.GetResponseStream());
var objText = reader.ReadToEnd(); // objText will have the value
}
}
I have to scrape a table from a secure site and I'm having trouble logging in to the page and retrieving the authentication token and any other associated cookies. Am I doing something wrong here?
public NameValueCollection LoginToDatrose()
{
var loginUriBuilder = new UriBuilder();
loginUriBuilder.Host = DatroseHostName;
loginUriBuilder.Path = BuildURIPath(DatroseBasePath, LOGIN_PAGE);
loginUriBuilder.Scheme = "https";
var boundary = Guid.NewGuid().ToString();
var postData = new NameValueCollection();
postData.Add("LoginName", DatroseUserName);
postData.Add("Password", DatrosePassword);
var data = Encoding.ASCII.GetBytes(postData.ToQueryString(false));
var request = WebRequest.Create(loginUriBuilder.Uri) as HttpWebRequest;
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded";
request.ContentLength = data.Length;
using (var d = request.GetRequestStream())
{
d.Write(data, 0, data.Length);
}
var response = request.GetResponse() as HttpWebResponse;
var responseCookies = new NameValueCollection();
foreach (var nvp in response.Cookies.OfType<Cookie>())
{
responseCookies.Add(nvp.Name, nvp.Value);
}
//using (var responseData = response.GetResponseStream())
//using (var responseReader = new StreamReader(responseData))
//{
// var theResponse = responseReader.ReadToEnd();
// Debug.WriteLine(theResponse);
//}
return responseCookies;
}
I get no values in the return object. It does not fail. The value of theResponse (when not commented out) seems to be the HTML of the login page.
Any assistance would be greatly appreciated.
OK, the problem here seems related to the 302 redirect that would occur after the credentials were passed. The HttpWebRequest would automatically follow the 302.
Ultimately, I ended up doing things a little differently. First, I subclassed the WebClient class as follows:
public class CookiesAwareWebClient : WebClient
{
private CookieContainer outboundCookies = new CookieContainer();
private CookieCollection inboundCookies = new CookieCollection();
public CookieContainer OutboundCookies
{
get
{
return outboundCookies;
}
}
public CookieCollection InboundCookies
{
get
{
return inboundCookies;
}
}
public bool IgnoreRedirects { get; set; }
protected override WebRequest GetWebRequest(Uri address)
{
WebRequest request = base.GetWebRequest(address);
if (request is HttpWebRequest)
{
(request as HttpWebRequest).CookieContainer = outboundCookies;
(request as HttpWebRequest).AllowAutoRedirect = !IgnoreRedirects;
}
return request;
}
protected override WebResponse GetWebResponse(WebRequest request)
{
WebResponse response = base.GetWebResponse(request);
if (response is HttpWebResponse)
{
inboundCookies = (response as HttpWebResponse).Cookies ?? inboundCookies;
}
return response;
}
}
This allowed me to use a WebClient class that was cookies-aware as well as one that I could control the redirect. Then I rewrote my code for logging in as follows:
public NameValueCollection LoginToDatrose()
{
var loginUriBuilder = new UriBuilder();
loginUriBuilder.Host = DatroseHostName;
loginUriBuilder.Path = BuildURIPath(DatroseBasePath, LOGIN_PAGE);
loginUriBuilder.Scheme = "https";
var postData = new NameValueCollection();
postData.Add("LoginName", DatroseUserName);
postData.Add("Password", DatrosePassword);
var responseCookies = new NameValueCollection();
using (var client = new CookiesAwareWebClient())
{
client.IgnoreRedirects = true;
var clientResponse = client.UploadValues(loginUriBuilder.Uri, "POST", postData);
foreach (var nvp in client.InboundCookies.OfType<Cookie>())
{
responseCookies.Add(nvp.Name, nvp.Value);
}
}
return responseCookies;
}
...and everything worked swimmingly.
I am working on a program that will log into a website and get certian data. However I am having trouble posting the login parameters and dealing with the cookies, as each time I get a page saying "You have logged out or Session has expired." So clearly I'm doing something wrong with posting the parameters or dealing with the cookies, but don't know which. I have been working on this for a while and just can't get my head around why this is not working correctly.
void Login2(string username, string password)
{
string pageSource;
string formUrl = "https://forUrl.com";
string formParams = string.Format("login={0}&sslProt={1}&pwd={2}&gru={3}", username, "", password, "115237091");
string cookieHeader;
HttpWebRequest req = (HttpWebRequest)WebRequest.Create(formUrl);
req.AllowAutoRedirect = false;
req.ContentType = "application/x-www-form-urlencoded";
req.Method = "POST";
byte[] bytes = System.Text.Encoding.ASCII.GetBytes(formParams);
req.ContentLength = bytes.Length;
using (Stream os = req.GetRequestStream())
{
os.Write(bytes, 0, bytes.Length);
}
HttpWebResponse resp = (HttpWebResponse)req.GetResponse();
cookieHeader = resp.Headers["Set-cookie"];
string getUrl = "https://Urlbehindform.com";
HttpWebRequest getRequest = (HttpWebRequest)WebRequest.Create(getUrl);
getRequest.Method = "GET";
getRequest.AllowAutoRedirect = false;
getRequest.Headers.Add("Cookie", cookieHeader);
HttpWebResponse getResponse = (HttpWebResponse)getRequest.GetResponse();
using (StreamReader sr = new StreamReader(getResponse.GetResponseStream()))
{
pageSource = sr.ReadToEnd();
}
Response.Redirect(getUrl);
}
I am getting the cookie when I do the POST and sending it back when I do the GET, but for some reason this doesn't seem to work. At first I thought it was the parameters, but after looking at the issue further using Tamper Data with Firefox the login parameters seem to be working fine. Any help would be great, as I have been working on this for a while and can't wrap my head around it. Thanks!
UPDATE:
After trying out a few suggestions I still can't get this to work. However Upon looking deeper into Data Tamper, It appears that there is a POST with the login parameters, then a GET to a different page and then finally the GET to the page after the login page (The one I'm trying to get to). After some further debugging I actually discovered that my login POST is not working as I thought, As the response header location is showing "/cv/scripts/A028/eng/logErr.asp". Meaning the rest of my code could have been fine all a long, it was that the POST wasn't giving me a valid login. Any Sugguestions as to why I am always getting the login error page? As always thanks for the help.
UPDATE:
After playing around further with Tamper Data is appears that the reason I am unable to get a successful login is that in order have a successful POST of the parameters there needs to be a cookie already obtained. How do I go about doing this?
Use a single CookieContainer for both requests. Then you don't have to copy cookies manually.
I [BMW1] added in a CookieContainer called cookies, but it still not working, Im not sure if im using the CookieContainer the right way. Here is an updated version of my code.
And edited by me [Hans Kesting], see comments with [HK]
void Login2(string username, string password)
{
string pageSource;
string formUrl = "https://server/cv/scripts/A028/eng/logProc.asp?ntry=0&dbg=";
string formParams = string.Format("login={0}&sslProt={1}&pwd={2}&gru={3}", username, "", password, "115237091");
// [HK] create a container for the cookies, where they are added automatically
CookieContainer cookies = new CookieContainer();
HttpWebRequest req = (HttpWebRequest)WebRequest.Create(formUrl);
req.CookieContainer = cookies;
req.AllowAutoRedirect = false;
req.ContentType = "application/x-www-form-urlencoded";
req.Method = "POST";
byte[] bytes = System.Text.Encoding.ASCII.GetBytes(formParams);
req.ContentLength = bytes.Length;
using (Stream os = req.GetRequestStream())
{
os.Write(bytes, 0, bytes.Length);
}
HttpWebResponse resp = (HttpWebResponse)req.GetResponse();
// [HK] no need to add cookies "by hand", that will happen automatically
//cookies.Add(resp.Cookies);
string getUrl = "https://server/cv/scripts/A028/eng/home.asp";
HttpWebRequest getRequest = (HttpWebRequest)WebRequest.Create(getUrl);
// [HK] use the same cookiecontainer as on the first request - correct
getRequest.CookieContainer = cookies;
getRequest.Method = "GET";
getRequest.AllowAutoRedirect = false;
HttpWebResponse getResponse = (HttpWebResponse)getRequest.GetResponse();
// [HK] no need to add cookies, they should be there already
//cookies.Add(getResponse.Cookies);
using (StreamReader sr = new StreamReader(getResponse.GetResponseStream()))
{
pageSource = sr.ReadToEnd();
}
// [HK] no need to add cookies, they should be there already
// cookies.Add(getResponse.Cookies);
Response.Redirect(getUrl);
}
You could use a Cookie aware web client,
public class CookieAwareWebClient : WebClient
{
public CookieContainer CookieContainer { get; set; }
public Uri Uri { get; set; }
public CookieAwareWebClient() : this (new CookieContainer())
{
}
public CookieAwareWebClient(CookieContainer cookies)
{
this.CookieContainer = cookies;
}
protected override WebRequest GetWebRequest(Uri address)
{
WebRequest request = base.GetWebRequest(address);
if (request is HttpWebRequest)
{
(request as HttpWebRequest).CookieContainer = this.CookieContainer;
}
HttpWebRequest httpRequest = (HttpWebRequest) request;
httpRequest.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
return httpRequest;
}
protected override WebResponse GetWebResponse(WebRequest request)
{
WebResponse response = base.GetWebResponse(request);
String setCookieHeader = response.Headers[HttpResponseHeader.SetCookie];
if (setCookieHeader != null)
{
//do something if needed to parse out the cookie.
if (setCookieHeader != null)
{
Cookie cookie = new Cookie(); //create cookie
this.CookieContainer.Add(cookie);
}
}
return response;
}
}
Example usage:
var wc = new CookieAwareWebClient ();
wc.Headers["Content-type"] = "application/x-www-form-urlencoded";
string HtmlResult = wc.UploadString(URI, myParameters);
I am logging into a page using HttpWebRequest and getting some information. I then use that information to create a new HttpWebRequest to get some more information. I do not want to use WebClient.
How can I pass the credentials I obtained from logging in using the first HttpWebRequest to the second one?
EDIT: If I use a CookieCollection then this is coming back as empty. I just tried using WebClient as a last resort and even for that it is not working, the second request takes me back to the login screen. I noticed that in a WebBrowser there is a cookie.
Add a CookieContainer to each request before you send it. Add the cookies you get from the first response to the second request. Assuming they use cookies for authentication, this should authenticate the second request.
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(urlWithParameters);
request.CookieContainer = new CookieContainer();
HttpWebResponse response = (HttpWebResponse) request.GetResponse();
var cookies = new CookieContainer();
cookies.Add( response.Cookies );
request = (HttpWebRequest)WebRequest.Create(secondUrlWithParameters);
request.CookieContainer = cookies;
...
this is just a sample running code based on answer 2. Maybe be redundant maybe illustrate somebody.
string url = "http://servername/place-where-data-is.extension"
string loginUrl = "https://servername/sampleLogin?email=eeeeee&passwd=xxxxxxx";
HttpWebRequest loginRequest = (HttpWebRequest)HttpWebRequest.Create(loginUrl);
loginRequest.CookieContainer = new CookieContainer();
loginRequest.Method = WebRequestMethods.Http.Get;
HttpWebResponse loginResponse = (HttpWebResponse)loginRequest.GetResponse();
var cookies = new CookieContainer();
cookies.Add(loginResponse.Cookies);
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(url);
request.CookieContainer = cookies;
request.Method = WebRequestMethods.Http.Get;
WebResponse response = (WebResponse)request.GetResponse();
Stream responseStream = response.GetResponseStream();
This is a really old question and I know it states no WebClient but I will post here for everyone who comes across this from Google. The original concept is not my code. I do not know where I originally found it.
using (WebClientEx client = new WebClientEx())
{
client.IntTimeout = intTimeout;
client.DownloadString(strReportUrlPrefix + strReportUrlQuery);
NameValueCollection auth = new NameValueCollection
{
{ "j_username", strReportUsername},
{ "j_password", strReportPassword}
};
byte[] data = client.UploadValues(strReportUrlPrefix + "j_security_check", auth);
// LOGIC HERE WITH DATA
}
WebClientEx Class:
public class WebClientEx : WebClient
{
private CookieContainer _cookieContainer = new CookieContainer();
public int IntTimeout { get; set; }
protected override WebRequest GetWebRequest(Uri address)
{
WebRequest request = base.GetWebRequest(address);
if (request != null)
request.Timeout = IntTimeout;
if (request is HttpWebRequest)
(request as HttpWebRequest).CookieContainer = _cookieContainer;
return request;
}
}
I am making a call to a page on my site using webclient. I'm trying to get the result of the webpage put into a pdf so I am trying to get a string representation of the rendered page. The problem is that the request is not authenticated so all I get is a login screen. I have sent the UseDefaultCredentials property to true but I still get the same result. Below is a portion of my code:
WebClient webClient = new WebClient();
webClient.Encoding = Encoding.UTF8;
webClient.UseDefaultCredentials = true;
return Encoding.UTF8.GetString(webClient.UploadValues(link, "POST",form));
You need to give the WebClient object the credentials. Something like this...
WebClient client = new WebClient();
client.Credentials = new NetworkCredential("username", "password");
What kind of authentication are you using? If it's Forms authentication, then at best, you'll have to find the .ASPXAUTH cookie and pass it in the WebClient request.
At worst, it won't work.
Public Function getWeb(ByRef sURL As String) As String
Dim myWebClient As New System.Net.WebClient()
Try
Dim myCredentialCache As New System.Net.CredentialCache()
Dim myURI As New Uri(sURL)
myCredentialCache.Add(myURI, "ntlm", System.Net.CredentialCache.DefaultNetworkCredentials)
myWebClient.Encoding = System.Text.Encoding.UTF8
myWebClient.Credentials = myCredentialCache
Return myWebClient.DownloadString(myURI)
Catch ex As Exception
Return "Exception " & ex.ToString()
End Try
End Function
This helped me to call API that was using cookie authentication. I have passed authorization in header like this:
request.Headers.Set("Authorization", Utility.Helper.ReadCookie("AuthCookie"));
complete code:
// utility method to read the cookie value:
public static string ReadCookie(string cookieName)
{
var cookies = HttpContext.Current.Request.Cookies;
var cookie = cookies.Get(cookieName);
if (cookie != null)
return cookie.Value;
return null;
}
// using statements where you are creating your webclient
using System.Web.Script.Serialization;
using System.Net;
using System.IO;
// WebClient:
var requestUrl = "<API_url>";
var postRequest = new ClassRoom { name = "kushal seth" };
using (var webClient = new WebClient()) {
JavaScriptSerializer serializer = new JavaScriptSerializer();
byte[] requestData = Encoding.ASCII.GetBytes(serializer.Serialize(postRequest));
HttpWebRequest request = WebRequest.Create(requestUrl) as HttpWebRequest;
request.Method = "POST";
request.ContentType = "application/json";
request.ContentLength = requestData.Length;
request.ContentType = "application/json";
request.Expect = "application/json";
request.Headers.Set("Authorization", Utility.Helper.ReadCookie("AuthCookie"));
request.GetRequestStream().Write(requestData, 0, requestData.Length);
using (var response = (HttpWebResponse)request.GetResponse()) {
var reader = new StreamReader(response.GetResponseStream());
var objText = reader.ReadToEnd(); // objText will have the value
}
}