Im using code like this to check if URL's are real and exist:
its working fine generally, but it's not working for youtube urls..
eg this valid url: http://www.youtube.com/watch?v=c4IBN5OAdZc
Utils.UrlExists(uri)
public static bool UrlExists(string url)
{
using (var client = new MyClient())
{
client.HeadOnly = true;
// fine, no content downloaded
try
{
string s1 = client.DownloadString(url);
return true;
}
catch { return false; }
}
}
class MyClient : WebClient
{
public bool HeadOnly { get; set; }
protected override WebRequest GetWebRequest(Uri address)
{
WebRequest req = base.GetWebRequest(address);
if (HeadOnly && req.Method == "GET")
{
req.Method = "HEAD";
}
return req;
}
}
Consider this approach:
Hit this URL with your C# code, and analyze the response:
http://gdata.youtube.com/feeds/api/videos/<your ID>
You will be able to see the HTTP response of 200 when successful.
Try it in your browser; you'll see that good IDs give you content, others will return plaintext of "Private video" or "Invalid id".
erics comment was the answer. If he puts an answer I'll mark it as correct. in the meantime I'll mark this as correct
Related
I've previously used a CookieContainer with HttpWebRequest and HttpWebResponse sessions, but now, I want to use it with a WebClient. As far as I understand, there is no built-in method like there is for HttpWebRequests (request.CookieContainer). How can I collect cookies from a WebClient in a CookieContainer?
I googled for this and found the following sample:
public class CookieAwareWebClient : WebClient
{
private readonly CookieContainer m_container = new CookieContainer();
protected override WebRequest GetWebRequest(Uri address)
{
WebRequest request = base.GetWebRequest(address);
HttpWebRequest webRequest = request as HttpWebRequest;
if (webRequest != null)
{
webRequest.CookieContainer = m_container;
}
return request;
}
}
Is this the best way to do it?
WebClient wb = new WebClient();
wb.Headers.Add(HttpRequestHeader.Cookie, "somecookie");
From Comments
How do you format the name and value of the cookie in place of "somecookie" ?
wb.Headers.Add(HttpRequestHeader.Cookie, "cookiename=cookievalue");
For multiple cookies:
wb.Headers.Add(HttpRequestHeader.Cookie,
"cookiename1=cookievalue1;" +
"cookiename2=cookievalue2");
Yes. IMHO, overriding GetWebRequest() is the best solution to WebClient's limited functionalty. Before I knew about this option, I wrote lots of really painful code at the HttpWebRequest layer because WebClient almost, but not quite, did what I needed. Derivation is much easier.
Another option is to use the regular WebClient class, but manually populate the Cookie header before making the request and then pull out the Set-Cookies header on the response. There are helper methods on the CookieContainer class which make creating and parsing these headers easier: CookieContainer.SetCookies() and CookieContainer.GetCookieHeader(), respectively.
I prefer the former approach since it's easier for the caller and requires less repetitive code than the second option. Also, the derivation approach works the same way for multiple extensibility scenarios (e.g. cookies, proxies, etc.).
This one is just extension of article you found.
public class WebClientEx : WebClient
{
public WebClientEx(CookieContainer container)
{
this.container = container;
}
public CookieContainer CookieContainer
{
get { return container; }
set { container= value; }
}
private CookieContainer container = new CookieContainer();
protected override WebRequest GetWebRequest(Uri address)
{
WebRequest r = base.GetWebRequest(address);
var request = r as HttpWebRequest;
if (request != null)
{
request.CookieContainer = container;
}
return r;
}
protected override WebResponse GetWebResponse(WebRequest request, IAsyncResult result)
{
WebResponse response = base.GetWebResponse(request, result);
ReadCookies(response);
return response;
}
protected override WebResponse GetWebResponse(WebRequest request)
{
WebResponse response = base.GetWebResponse(request);
ReadCookies(response);
return response;
}
private void ReadCookies(WebResponse r)
{
var response = r as HttpWebResponse;
if (response != null)
{
CookieCollection cookies = response.Cookies;
container.Add(cookies);
}
}
}
The HttpWebRequest modifies the CookieContainer assigned to it. There is no need to process returned cookies. Simply assign your cookie container to every web request.
public class CookieAwareWebClient : WebClient
{
public CookieContainer CookieContainer { get; set; } = new CookieContainer();
protected override WebRequest GetWebRequest(Uri uri)
{
WebRequest request = base.GetWebRequest(uri);
if (request is HttpWebRequest)
{
(request as HttpWebRequest).CookieContainer = CookieContainer;
}
return request;
}
}
I think there's cleaner way where you don't have to create a new webclient (and it'll work with 3rd party libraries as well)
internal static class MyWebRequestCreator
{
private static IWebRequestCreate myCreator;
public static IWebRequestCreate MyHttp
{
get
{
if (myCreator == null)
{
myCreator = new MyHttpRequestCreator();
}
return myCreator;
}
}
private class MyHttpRequestCreator : IWebRequestCreate
{
public WebRequest Create(Uri uri)
{
var req = System.Net.WebRequest.CreateHttp(uri);
req.CookieContainer = new CookieContainer();
return req;
}
}
}
Now all you have to do is opt in for which domains you want to use this:
WebRequest.RegisterPrefix("http://example.com/", MyWebRequestCreator.MyHttp);
That means ANY webrequest that goes to example.com will now use your custom webrequest creator, including the standard webclient. This approach means you don't have to touch all you code. You just call the register prefix once and be done with it.
You can also register for "http" prefix to opt in for everything everywhere.
I'm having trouble checking if a url exists.
I use the below for most checks - its the best method I have found and saves on a full web request but it does;nt allow for checks on address such as:
m.bbc.co.uk
Any mobile site for m., has no effect and breaks.
public static bool Does_URL_Exists(string str_url)
{
// using MyClient from linked post
using (var client = new MyClient())
{
client.HeadOnly = true;
// fine, no content downloaded
try
{
//System.Windows.Forms.MessageBox.Show(str_url);
string s1 = client.DownloadString(str_url);
return true;
}
catch
{
return false;
}
}
}
class MyClient : WebClient
{
public bool HeadOnly { get; set; }
protected override WebRequest GetWebRequest(Uri address)
{
WebRequest req = base.GetWebRequest(address);
if (HeadOnly && req.Method == "GET")
{
req.Method = "HEAD";
}
return req;
}
}
Any clues on how I cam make this work. www.bbc.co.uk/m is no good either.
you don't need load all string of link to client, only check status when connect to url is OK (200).
with url as: "http://www.bbc.co.uk/". i think mobile like of direct link is: "http://www.m.bbc.co.uk/"
Hope help to you
I have a situation when the source I get from a site using WebClient/HttpWebRequest is different from the actual source from web browsing (different button_onclick).
I guess the site updates the button very short after sending the first source,
I would like to get an updated source of a site after like 1 second I've "been" there.
I've tried something, but I don't really know how to do that.
This is my try:
public class KeepAliveWebClient : WebClient
{
public string DownloadString(string address)
{
return base.DownloadString(address);
}
protected override WebRequest GetWebRequest(Uri address)
{
HttpWebRequest request = (HttpWebRequest)base.GetWebRequest(address);
request.KeepAlive = true;
return request;
}
protected override WebResponse GetWebResponse(WebRequest request)
{
var response = base.GetWebResponse(request);
Thread.Sleep(2000);
var newResponse = base.GetWebResponse(request);
return newResponse;
}
}
Thanks in advance.
I've previously used a CookieContainer with HttpWebRequest and HttpWebResponse sessions, but now, I want to use it with a WebClient. As far as I understand, there is no built-in method like there is for HttpWebRequests (request.CookieContainer). How can I collect cookies from a WebClient in a CookieContainer?
I googled for this and found the following sample:
public class CookieAwareWebClient : WebClient
{
private readonly CookieContainer m_container = new CookieContainer();
protected override WebRequest GetWebRequest(Uri address)
{
WebRequest request = base.GetWebRequest(address);
HttpWebRequest webRequest = request as HttpWebRequest;
if (webRequest != null)
{
webRequest.CookieContainer = m_container;
}
return request;
}
}
Is this the best way to do it?
WebClient wb = new WebClient();
wb.Headers.Add(HttpRequestHeader.Cookie, "somecookie");
From Comments
How do you format the name and value of the cookie in place of "somecookie" ?
wb.Headers.Add(HttpRequestHeader.Cookie, "cookiename=cookievalue");
For multiple cookies:
wb.Headers.Add(HttpRequestHeader.Cookie,
"cookiename1=cookievalue1;" +
"cookiename2=cookievalue2");
Yes. IMHO, overriding GetWebRequest() is the best solution to WebClient's limited functionalty. Before I knew about this option, I wrote lots of really painful code at the HttpWebRequest layer because WebClient almost, but not quite, did what I needed. Derivation is much easier.
Another option is to use the regular WebClient class, but manually populate the Cookie header before making the request and then pull out the Set-Cookies header on the response. There are helper methods on the CookieContainer class which make creating and parsing these headers easier: CookieContainer.SetCookies() and CookieContainer.GetCookieHeader(), respectively.
I prefer the former approach since it's easier for the caller and requires less repetitive code than the second option. Also, the derivation approach works the same way for multiple extensibility scenarios (e.g. cookies, proxies, etc.).
This one is just extension of article you found.
public class WebClientEx : WebClient
{
public WebClientEx(CookieContainer container)
{
this.container = container;
}
public CookieContainer CookieContainer
{
get { return container; }
set { container= value; }
}
private CookieContainer container = new CookieContainer();
protected override WebRequest GetWebRequest(Uri address)
{
WebRequest r = base.GetWebRequest(address);
var request = r as HttpWebRequest;
if (request != null)
{
request.CookieContainer = container;
}
return r;
}
protected override WebResponse GetWebResponse(WebRequest request, IAsyncResult result)
{
WebResponse response = base.GetWebResponse(request, result);
ReadCookies(response);
return response;
}
protected override WebResponse GetWebResponse(WebRequest request)
{
WebResponse response = base.GetWebResponse(request);
ReadCookies(response);
return response;
}
private void ReadCookies(WebResponse r)
{
var response = r as HttpWebResponse;
if (response != null)
{
CookieCollection cookies = response.Cookies;
container.Add(cookies);
}
}
}
The HttpWebRequest modifies the CookieContainer assigned to it. There is no need to process returned cookies. Simply assign your cookie container to every web request.
public class CookieAwareWebClient : WebClient
{
public CookieContainer CookieContainer { get; set; } = new CookieContainer();
protected override WebRequest GetWebRequest(Uri uri)
{
WebRequest request = base.GetWebRequest(uri);
if (request is HttpWebRequest)
{
(request as HttpWebRequest).CookieContainer = CookieContainer;
}
return request;
}
}
I think there's cleaner way where you don't have to create a new webclient (and it'll work with 3rd party libraries as well)
internal static class MyWebRequestCreator
{
private static IWebRequestCreate myCreator;
public static IWebRequestCreate MyHttp
{
get
{
if (myCreator == null)
{
myCreator = new MyHttpRequestCreator();
}
return myCreator;
}
}
private class MyHttpRequestCreator : IWebRequestCreate
{
public WebRequest Create(Uri uri)
{
var req = System.Net.WebRequest.CreateHttp(uri);
req.CookieContainer = new CookieContainer();
return req;
}
}
}
Now all you have to do is opt in for which domains you want to use this:
WebRequest.RegisterPrefix("http://example.com/", MyWebRequestCreator.MyHttp);
That means ANY webrequest that goes to example.com will now use your custom webrequest creator, including the standard webclient. This approach means you don't have to touch all you code. You just call the register prefix once and be done with it.
You can also register for "http" prefix to opt in for everything everywhere.
I've previously used a CookieContainer with HttpWebRequest and HttpWebResponse sessions, but now, I want to use it with a WebClient. As far as I understand, there is no built-in method like there is for HttpWebRequests (request.CookieContainer). How can I collect cookies from a WebClient in a CookieContainer?
I googled for this and found the following sample:
public class CookieAwareWebClient : WebClient
{
private readonly CookieContainer m_container = new CookieContainer();
protected override WebRequest GetWebRequest(Uri address)
{
WebRequest request = base.GetWebRequest(address);
HttpWebRequest webRequest = request as HttpWebRequest;
if (webRequest != null)
{
webRequest.CookieContainer = m_container;
}
return request;
}
}
Is this the best way to do it?
WebClient wb = new WebClient();
wb.Headers.Add(HttpRequestHeader.Cookie, "somecookie");
From Comments
How do you format the name and value of the cookie in place of "somecookie" ?
wb.Headers.Add(HttpRequestHeader.Cookie, "cookiename=cookievalue");
For multiple cookies:
wb.Headers.Add(HttpRequestHeader.Cookie,
"cookiename1=cookievalue1;" +
"cookiename2=cookievalue2");
Yes. IMHO, overriding GetWebRequest() is the best solution to WebClient's limited functionalty. Before I knew about this option, I wrote lots of really painful code at the HttpWebRequest layer because WebClient almost, but not quite, did what I needed. Derivation is much easier.
Another option is to use the regular WebClient class, but manually populate the Cookie header before making the request and then pull out the Set-Cookies header on the response. There are helper methods on the CookieContainer class which make creating and parsing these headers easier: CookieContainer.SetCookies() and CookieContainer.GetCookieHeader(), respectively.
I prefer the former approach since it's easier for the caller and requires less repetitive code than the second option. Also, the derivation approach works the same way for multiple extensibility scenarios (e.g. cookies, proxies, etc.).
This one is just extension of article you found.
public class WebClientEx : WebClient
{
public WebClientEx(CookieContainer container)
{
this.container = container;
}
public CookieContainer CookieContainer
{
get { return container; }
set { container= value; }
}
private CookieContainer container = new CookieContainer();
protected override WebRequest GetWebRequest(Uri address)
{
WebRequest r = base.GetWebRequest(address);
var request = r as HttpWebRequest;
if (request != null)
{
request.CookieContainer = container;
}
return r;
}
protected override WebResponse GetWebResponse(WebRequest request, IAsyncResult result)
{
WebResponse response = base.GetWebResponse(request, result);
ReadCookies(response);
return response;
}
protected override WebResponse GetWebResponse(WebRequest request)
{
WebResponse response = base.GetWebResponse(request);
ReadCookies(response);
return response;
}
private void ReadCookies(WebResponse r)
{
var response = r as HttpWebResponse;
if (response != null)
{
CookieCollection cookies = response.Cookies;
container.Add(cookies);
}
}
}
The HttpWebRequest modifies the CookieContainer assigned to it. There is no need to process returned cookies. Simply assign your cookie container to every web request.
public class CookieAwareWebClient : WebClient
{
public CookieContainer CookieContainer { get; set; } = new CookieContainer();
protected override WebRequest GetWebRequest(Uri uri)
{
WebRequest request = base.GetWebRequest(uri);
if (request is HttpWebRequest)
{
(request as HttpWebRequest).CookieContainer = CookieContainer;
}
return request;
}
}
I think there's cleaner way where you don't have to create a new webclient (and it'll work with 3rd party libraries as well)
internal static class MyWebRequestCreator
{
private static IWebRequestCreate myCreator;
public static IWebRequestCreate MyHttp
{
get
{
if (myCreator == null)
{
myCreator = new MyHttpRequestCreator();
}
return myCreator;
}
}
private class MyHttpRequestCreator : IWebRequestCreate
{
public WebRequest Create(Uri uri)
{
var req = System.Net.WebRequest.CreateHttp(uri);
req.CookieContainer = new CookieContainer();
return req;
}
}
}
Now all you have to do is opt in for which domains you want to use this:
WebRequest.RegisterPrefix("http://example.com/", MyWebRequestCreator.MyHttp);
That means ANY webrequest that goes to example.com will now use your custom webrequest creator, including the standard webclient. This approach means you don't have to touch all you code. You just call the register prefix once and be done with it.
You can also register for "http" prefix to opt in for everything everywhere.