I have the following code:
What do I put as the second argument for GetPage?
The second argument should be the previous cookie request of the same url.
for example I put google.com as the first argument to get the cookie when I make a get Request, but for the second how do I insert it?
static void Main()
{
GetPage("http://google.com/",cookieContainer??);
}
public class CookieAwareWebClient : WebClient
{
public CookieAwareWebClient(CookieContainer container)
{
CookieContainer = container;
}
public CookieAwareWebClient()
: this(new CookieContainer())
{ }
public CookieContainer CookieContainer { get; private set; }
protected override WebRequest GetWebRequest(Uri address)
{
var request = (HttpWebRequest)base.GetWebRequest(address);
request.CookieContainer = CookieContainer;
return request;
}
}
public HtmlAgilityPack.HtmlDocument GetPage(string url, CookieContainer CookieContainer)
{
Uri absoluteUri = new Uri("http://google.com/");
var cookies = CookieContainer.GetCookies(absoluteUri);
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.CookieContainer = new CookieContainer();
foreach (Cookie cookie in cookies)
{
request.CookieContainer.Add(cookie);
}
request.Method = "GET";
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
var stream = response.GetResponseStream();
using (var reader = new StreamReader(stream))
{
string html = reader.ReadToEnd();
var doc = new HtmlAgilityPack.HtmlDocument();
doc.LoadHtml(html);
return doc;
}
}
Related
I login with HTTPWebRequest and transfer the CookieContainer to the new class and want my WebClient to log in with this cookie and download a site as string. But I only get the source code of the login page, so it seems like the webclient won't login.
This is my current code:
BasicAuthentication login = new BasicAuthentication();
CookieContainer cookieJar = login.getCookie();
WebClientEx client = new WebClientEx();
client.Cookies = cookieJar;
And the WebClientEx Class
class WebClientEx : WebClient
{
private CookieContainer _cookies;
private string _ref;
public WebClientEx()
{
_cookies = new CookieContainer();
}
public CookieContainer Cookies
{
get { return _cookies; }
set { _cookies = value; }
}
protected override WebRequest GetWebRequest(System.Uri address)
{
var webReq = base.GetWebRequest(address);
if (webReq is HttpWebRequest)
{
var req = (HttpWebRequest)webReq;
req.CookieContainer = _cookies;
if (_ref != null)
{
req.Referer = _ref;
}
}
_ref = address.ToString();
return webReq;
}
protected override void Dispose(bool disposing)
{
_cookies = null;
base.Dispose(disposing);
}
}
I just get an Answer to my question.
Here's a example of my code:
WebClient client = new WebClient();
client.Headers.Add(HttpRequestHeader.Cookie, "cookievalue");
string download = client.DownloadString("url");
output.Text = download;
In the variable cookievalue I only enter the cookiename=cookievalue.
I get the name and the value from Postman, a useful google chrome tool.
I just have to make sure that i can redirect to any login proteccted page with postman and when this is true, then I can use cookiename and cookievalue to login my webclient and see all pages and download sites with downloadString.
I've previously used a CookieContainer with HttpWebRequest and HttpWebResponse sessions, but now, I want to use it with a WebClient. As far as I understand, there is no built-in method like there is for HttpWebRequests (request.CookieContainer). How can I collect cookies from a WebClient in a CookieContainer?
I googled for this and found the following sample:
public class CookieAwareWebClient : WebClient
{
private readonly CookieContainer m_container = new CookieContainer();
protected override WebRequest GetWebRequest(Uri address)
{
WebRequest request = base.GetWebRequest(address);
HttpWebRequest webRequest = request as HttpWebRequest;
if (webRequest != null)
{
webRequest.CookieContainer = m_container;
}
return request;
}
}
Is this the best way to do it?
WebClient wb = new WebClient();
wb.Headers.Add(HttpRequestHeader.Cookie, "somecookie");
From Comments
How do you format the name and value of the cookie in place of "somecookie" ?
wb.Headers.Add(HttpRequestHeader.Cookie, "cookiename=cookievalue");
For multiple cookies:
wb.Headers.Add(HttpRequestHeader.Cookie,
"cookiename1=cookievalue1;" +
"cookiename2=cookievalue2");
Yes. IMHO, overriding GetWebRequest() is the best solution to WebClient's limited functionalty. Before I knew about this option, I wrote lots of really painful code at the HttpWebRequest layer because WebClient almost, but not quite, did what I needed. Derivation is much easier.
Another option is to use the regular WebClient class, but manually populate the Cookie header before making the request and then pull out the Set-Cookies header on the response. There are helper methods on the CookieContainer class which make creating and parsing these headers easier: CookieContainer.SetCookies() and CookieContainer.GetCookieHeader(), respectively.
I prefer the former approach since it's easier for the caller and requires less repetitive code than the second option. Also, the derivation approach works the same way for multiple extensibility scenarios (e.g. cookies, proxies, etc.).
This one is just extension of article you found.
public class WebClientEx : WebClient
{
public WebClientEx(CookieContainer container)
{
this.container = container;
}
public CookieContainer CookieContainer
{
get { return container; }
set { container= value; }
}
private CookieContainer container = new CookieContainer();
protected override WebRequest GetWebRequest(Uri address)
{
WebRequest r = base.GetWebRequest(address);
var request = r as HttpWebRequest;
if (request != null)
{
request.CookieContainer = container;
}
return r;
}
protected override WebResponse GetWebResponse(WebRequest request, IAsyncResult result)
{
WebResponse response = base.GetWebResponse(request, result);
ReadCookies(response);
return response;
}
protected override WebResponse GetWebResponse(WebRequest request)
{
WebResponse response = base.GetWebResponse(request);
ReadCookies(response);
return response;
}
private void ReadCookies(WebResponse r)
{
var response = r as HttpWebResponse;
if (response != null)
{
CookieCollection cookies = response.Cookies;
container.Add(cookies);
}
}
}
The HttpWebRequest modifies the CookieContainer assigned to it. There is no need to process returned cookies. Simply assign your cookie container to every web request.
public class CookieAwareWebClient : WebClient
{
public CookieContainer CookieContainer { get; set; } = new CookieContainer();
protected override WebRequest GetWebRequest(Uri uri)
{
WebRequest request = base.GetWebRequest(uri);
if (request is HttpWebRequest)
{
(request as HttpWebRequest).CookieContainer = CookieContainer;
}
return request;
}
}
I think there's cleaner way where you don't have to create a new webclient (and it'll work with 3rd party libraries as well)
internal static class MyWebRequestCreator
{
private static IWebRequestCreate myCreator;
public static IWebRequestCreate MyHttp
{
get
{
if (myCreator == null)
{
myCreator = new MyHttpRequestCreator();
}
return myCreator;
}
}
private class MyHttpRequestCreator : IWebRequestCreate
{
public WebRequest Create(Uri uri)
{
var req = System.Net.WebRequest.CreateHttp(uri);
req.CookieContainer = new CookieContainer();
return req;
}
}
}
Now all you have to do is opt in for which domains you want to use this:
WebRequest.RegisterPrefix("http://example.com/", MyWebRequestCreator.MyHttp);
That means ANY webrequest that goes to example.com will now use your custom webrequest creator, including the standard webclient. This approach means you don't have to touch all you code. You just call the register prefix once and be done with it.
You can also register for "http" prefix to opt in for everything everywhere.
Well really I guess it would be https://login.live.com/, But regardless my goal is to make a windows application that will be able to log into bing.com. Right now I have a button on my windows form that's purpose is to log into bing.com. I found some code on how to log into websites and tried to implement it but to no avail. Right now this is what I have in the button click event:
var loginAddress = "https://login.live.com/";
var loginData = new NameValueCollection
{
{ "username", "myUserName" },
{ "password", "myPassword" }
};
var client = new CookieAwareWebClient();
client.Login(loginAddress, loginData);
Process.Start("IExplore.exe", "https://login.live.com/");
Here is my CookieAwareWebClient class code:
public class CookieAwareWebClient : WebClient
{
public void Login(string loginPageAddress, NameValueCollection loginData)
{
CookieContainer container;
var request = (HttpWebRequest)WebRequest.Create(loginPageAddress);
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded";
var buffer = Encoding.ASCII.GetBytes(loginData.ToString());
request.ContentLength = buffer.Length;
var requestStream = request.GetRequestStream();
requestStream.Write(buffer, 0, buffer.Length);
requestStream.Close();
container = request.CookieContainer = new CookieContainer();
var response = request.GetResponse();
response.Close();
CookieContainer = container;
}
public CookieAwareWebClient(CookieContainer container)
{
CookieContainer = container;
}
public CookieAwareWebClient()
: this(new CookieContainer())
{ }
public CookieContainer CookieContainer { get; private set; }
protected override WebRequest GetWebRequest(Uri address)
{
var request = (HttpWebRequest)base.GetWebRequest(address);
request.CookieContainer = CookieContainer;
return request;
}
}
This question already has answers here:
Using CookieContainer with WebClient class
(5 answers)
Accept Cookies in WebClient?
(3 answers)
Closed 8 years ago.
I logged site by httprequest and cookie in varible cookiecontanse . But now i need to set cookie got for webclient . I used line
webClient.Headers.Add(HttpRequestHeader.Cookie,cookiecontanse.ToString)
But not work. Why ?
Usage :
CookieContainer cookieJar = new CookieContainer();
cookieJar.Add(new Cookie("my_cookie", "cookie_value", "/", "mysite"));
CookieAwareWebClient client = new CookieAwareWebClient(cookieJar);
string response = client.DownloadString("http://example.com/response_with_cookie_only.php");
public class CookieAwareWebClient : WebClient
{
public CookieContainer CookieContainer { get; set; }
public Uri Uri { get; set; }
public CookieAwareWebClient()
: this(new CookieContainer())
{
}
public CookieAwareWebClient(CookieContainer cookies)
{
this.CookieContainer = cookies;
}
protected override WebRequest GetWebRequest(Uri address)
{
WebRequest request = base.GetWebRequest(address);
if (request is HttpWebRequest)
{
(request as HttpWebRequest).CookieContainer = this.CookieContainer;
}
HttpWebRequest httpRequest = (HttpWebRequest)request;
httpRequest.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
return httpRequest;
}
protected override WebResponse GetWebResponse(WebRequest request)
{
WebResponse response = base.GetWebResponse(request);
String setCookieHeader = response.Headers[HttpResponseHeader.SetCookie];
if (setCookieHeader != null)
{
//do something if needed to parse out the cookie.
if (setCookieHeader != null)
{
Cookie cookie = new Cookie(); //create cookie
this.CookieContainer.Add(cookie);
}
}
return response;
}
}
You will see two overridden methods for GetWebRequest and GetWebResponse. These methods can be overridden to handle the cookie container.
I've previously used a CookieContainer with HttpWebRequest and HttpWebResponse sessions, but now, I want to use it with a WebClient. As far as I understand, there is no built-in method like there is for HttpWebRequests (request.CookieContainer). How can I collect cookies from a WebClient in a CookieContainer?
I googled for this and found the following sample:
public class CookieAwareWebClient : WebClient
{
private readonly CookieContainer m_container = new CookieContainer();
protected override WebRequest GetWebRequest(Uri address)
{
WebRequest request = base.GetWebRequest(address);
HttpWebRequest webRequest = request as HttpWebRequest;
if (webRequest != null)
{
webRequest.CookieContainer = m_container;
}
return request;
}
}
Is this the best way to do it?
WebClient wb = new WebClient();
wb.Headers.Add(HttpRequestHeader.Cookie, "somecookie");
From Comments
How do you format the name and value of the cookie in place of "somecookie" ?
wb.Headers.Add(HttpRequestHeader.Cookie, "cookiename=cookievalue");
For multiple cookies:
wb.Headers.Add(HttpRequestHeader.Cookie,
"cookiename1=cookievalue1;" +
"cookiename2=cookievalue2");
Yes. IMHO, overriding GetWebRequest() is the best solution to WebClient's limited functionalty. Before I knew about this option, I wrote lots of really painful code at the HttpWebRequest layer because WebClient almost, but not quite, did what I needed. Derivation is much easier.
Another option is to use the regular WebClient class, but manually populate the Cookie header before making the request and then pull out the Set-Cookies header on the response. There are helper methods on the CookieContainer class which make creating and parsing these headers easier: CookieContainer.SetCookies() and CookieContainer.GetCookieHeader(), respectively.
I prefer the former approach since it's easier for the caller and requires less repetitive code than the second option. Also, the derivation approach works the same way for multiple extensibility scenarios (e.g. cookies, proxies, etc.).
This one is just extension of article you found.
public class WebClientEx : WebClient
{
public WebClientEx(CookieContainer container)
{
this.container = container;
}
public CookieContainer CookieContainer
{
get { return container; }
set { container= value; }
}
private CookieContainer container = new CookieContainer();
protected override WebRequest GetWebRequest(Uri address)
{
WebRequest r = base.GetWebRequest(address);
var request = r as HttpWebRequest;
if (request != null)
{
request.CookieContainer = container;
}
return r;
}
protected override WebResponse GetWebResponse(WebRequest request, IAsyncResult result)
{
WebResponse response = base.GetWebResponse(request, result);
ReadCookies(response);
return response;
}
protected override WebResponse GetWebResponse(WebRequest request)
{
WebResponse response = base.GetWebResponse(request);
ReadCookies(response);
return response;
}
private void ReadCookies(WebResponse r)
{
var response = r as HttpWebResponse;
if (response != null)
{
CookieCollection cookies = response.Cookies;
container.Add(cookies);
}
}
}
The HttpWebRequest modifies the CookieContainer assigned to it. There is no need to process returned cookies. Simply assign your cookie container to every web request.
public class CookieAwareWebClient : WebClient
{
public CookieContainer CookieContainer { get; set; } = new CookieContainer();
protected override WebRequest GetWebRequest(Uri uri)
{
WebRequest request = base.GetWebRequest(uri);
if (request is HttpWebRequest)
{
(request as HttpWebRequest).CookieContainer = CookieContainer;
}
return request;
}
}
I think there's cleaner way where you don't have to create a new webclient (and it'll work with 3rd party libraries as well)
internal static class MyWebRequestCreator
{
private static IWebRequestCreate myCreator;
public static IWebRequestCreate MyHttp
{
get
{
if (myCreator == null)
{
myCreator = new MyHttpRequestCreator();
}
return myCreator;
}
}
private class MyHttpRequestCreator : IWebRequestCreate
{
public WebRequest Create(Uri uri)
{
var req = System.Net.WebRequest.CreateHttp(uri);
req.CookieContainer = new CookieContainer();
return req;
}
}
}
Now all you have to do is opt in for which domains you want to use this:
WebRequest.RegisterPrefix("http://example.com/", MyWebRequestCreator.MyHttp);
That means ANY webrequest that goes to example.com will now use your custom webrequest creator, including the standard webclient. This approach means you don't have to touch all you code. You just call the register prefix once and be done with it.
You can also register for "http" prefix to opt in for everything everywhere.