We Keep Coding

c# Reading source code with HttpWebRequest

My code is as below. But I am getting error. How can I read the url source code in the example.
Error Message 'The remote server returned an error: (403) Forbidden'
CookieContainer cookierJar = new CookieContainer();
HttpWebRequest webRequest = (HttpWebRequest)WebRequest.Create("https://www.hepsiburada.com/dyson-v11-absolute-extra-kablosuz-supurge-dyson-turkiye-garantili-p-HBV00000W4W6U");
webRequest.AllowAutoRedirect = true;
webRequest.CachePolicy = new System.Net.Cache.RequestCachePolicy(System.Net.Cache.RequestCacheLevel.NoCacheNoStore);
webRequest.CookieContainer = cookierJar;
webRequest.UserAgent = "My Thirsty Browser";
int statusCode = 0;
string sourceCode = string.Empty;
try
{
HttpWebResponse webResponse = (HttpWebResponse)webRequest.GetResponse();
statusCode = (int)webResponse.StatusCode;
StreamReader readContent = new StreamReader(webResponse.GetResponseStream());
sourceCode = readContent.ReadToEnd();
webResponse.Close();
webResponse = null;
}
catch (WebException xc)
{
if (xc.Response is HttpWebResponse)
{
HttpWebResponse rs = xc.Response as HttpWebResponse;
StreamReader readContent = new StreamReader(rs.GetResponseStream());
if (readContent != null)
{
sourceCode = readContent.ReadToEnd();
}
statusCode = (int)rs.StatusCode;
}
else
{
statusCode = (int)xc.Status;
sourceCode = xc.Message;
}
}
catch (Exception xc)
{
sourceCode = xc.Message;
}

I get a 403 in IE, I guess you need to be logged in to retrieve the resource. Your browser may have the credentials cached but your app isn't designed to log you in. Or are you logged in to Google in your browser - try logging out and see if you still have access....

Return full HTTP Response in ASP.Net WebAPI Controller

I'm trying to return a full HTTP-Response to the browser within an ASP.NET WebAPI Controller.
The scenario is the following:
I make a remote call to another webserver and get a full HTTP-Message including the HTTP Headers and content. I just want do deliver this message "as is" to the browser.
Does anyone know how to do this?

Create your own IHttpHandler and configure a route for it. You have to copy all response headers from your own response to the response object of ASP.NET.
Here is a sample implementation for another scenario:
public class CorsProxyHttpHandler : IHttpHandler
{
public void ProcessRequest(HttpContext context)
{
var url = context.Request.Headers["X-CorsProxy-Url"];
if (url == null)
{
context.Response.StatusCode = 501;
context.Response.StatusDescription =
"X-CorsProxy-Url was not specified. The corsproxy should only be invoked from the proxy JavaScript.";
context.Response.End();
return;
}
try
{
var request = WebRequest.CreateHttp(url);
context.Request.CopyHeadersTo(request);
request.Method = context.Request.HttpMethod;
request.ContentType = context.Request.ContentType;
request.UserAgent = context.Request.UserAgent;
if (context.Request.AcceptTypes != null)
request.Accept = string.Join(";", context.Request.AcceptTypes);
if (context.Request.UrlReferrer != null)
request.Referer = context.Request.UrlReferrer.ToString();
if (!context.Request.HttpMethod.Equals("GET", StringComparison.OrdinalIgnoreCase))
context.Request.InputStream.CopyTo(request.GetRequestStream());
var response = (HttpWebResponse)request.GetResponse();
response.CopyHeadersTo(context.Response);
context.Response.ContentType = response.ContentType;
context.Response.StatusCode =(int) response.StatusCode;
context.Response.StatusDescription = response.StatusDescription;
var stream = response.GetResponseStream();
if (stream != null && response.ContentLength > 0)
{
stream.CopyTo(context.Response.OutputStream);
stream.Flush();
}
}
catch (WebException exception)
{
context.Response.AddHeader("X-CorsProxy-InternalFailure", "false");
var response = exception.Response as HttpWebResponse;
if (response != null)
{
context.Response.StatusCode = (int)response.StatusCode;
context.Response.StatusDescription = response.StatusDescription;
response.CopyHeadersTo(context.Response);
var stream = response.GetResponseStream();
if (stream != null)
stream.CopyTo(context.Response.OutputStream);
return;
}
context.Response.StatusCode = 501;
context.Response.StatusDescription = exception.Status.ToString();
var msg = Encoding.ASCII.GetBytes(exception.Message);
context.Response.OutputStream.Write(msg, 0, msg.Length);
context.Response.Close();
}
catch (Exception exception)
{
context.Response.StatusCode = 501;
context.Response.StatusDescription = "Failed to call proxied url.";
context.Response.AddHeader("X-CorsProxy-InternalFailure", "true");
var msg = Encoding.ASCII.GetBytes(exception.Message);
context.Response.OutputStream.Write(msg, 0, msg.Length);
context.Response.Close();
}
}
public bool IsReusable { get { return true; }}
}
(from my article: http://blog.gauffin.org/2014/04/how-to-use-cors-requests-in-internet-explorer-9-and-below/)

HttpWebRequest in wpf

im making an app in wpf, which uses a restful api, from sharefile im making the autorisation
so far i have this
private void Button_Click_1(object sender, RoutedEventArgs e)
{
Uri uri = new Uri("https://secure.sharefile.com/oauth/authorize");
HttpWebRequest request = WebRequest.CreateHttp(uri);
shareFileWebView.Navigate(uri);
request.Method = "POST";
shareFileWebView. // but i supose to get something from here
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Console.WriteLine(response.StatusCode);
JObject token = null;
using (var reader = new StreamReader(response.GetResponseStream()))
{
string body = reader.ReadToEnd();
token = JObject.Parse(body);
}
OAuth2Token _tokene =new OAuth2Token(token);
}
i need to get that token but how can i get from the webbrowser? in the webr browser the user fills his accounts data, any ideas??

Does this link give you what you need?
http://api.sharefile.com/rest/api-key.aspx

Http request with body ( for example send image ) :
public string httpRequest(string url,byte[] image)
{
try
{
var request = (HttpWebRequest)WebRequest.Create(url);
request.Method = "Post";
request.ContentType = "multipart/form-data";
request.ContentLength = image.Length;
using (Stream postStream = request.GetRequestStream())
{
postStream.Write(image, 0, image.Length);
postStream.Close();
WebResponse response = request.GetResponse();
var responseString = new StreamReader(response.GetResponseStream()).ReadToEnd();
if (!string.IsNullOrEmpty(responseString))
return responseString;
return null;
}
}
catch (Exception ex)
{
return ex.Message;
}
}
or :
public string RunCommand()
{
ServicePointManager.ServerCertificateValidationCallback += (sender, cert, chain, sslPolicyErrors) => true;
var request = (HttpWebRequest)WebRequest.Create("url");
var response = (HttpWebResponse)request.GetResponse();
var responseString = new StreamReader(response.GetResponseStream()).ReadToEnd();
response.Close();
return responseString;
}

Check to see if site is online, limit the timeout

I'm trying to make a function that checks if a site is online or not, but is having some problem with the timeout. I want to limit it to a max 3 sec, if there is no respons within 3 sec I should see the page as offline.
My try:
class OnlineCheck
{
public static bool IsOnline(string url)
{
try
{
WebClient webclient = new WebClient();
webclient.Headers.Add(HttpRequestHeader.KeepAlive, "1000");
webclient.OpenRead(url);
}
catch { return false; }
return true;
}
}

The WebClient doesn't support timeout. But you can use the HttpWebRequest!
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(Endpoint);
request.Timeout=3000;
request.GetResponse();

If you want to check that the site is online, you are not really interested in the content of the page, just that you get a response. To make that more efficient, you should only request the http headers. Here is a quick example on how you could do:
private static IEnumerable<HttpStatusCode> onlineStatusCodes = new[]
{
HttpStatusCode.Accepted,
HttpStatusCode.Found,
HttpStatusCode.OK,
// add more codes as needed
};
private static bool IsSiteOnline(string url, int timeout)
{
HttpWebRequest request = WebRequest.Create(url) as HttpWebRequest;
{
if (request != null)
{
request.Method = "HEAD"; // get headers only
request.Timeout = timeout;
using (var response = request.GetResponse() as HttpWebResponse)
{
return response != null && onlineStatusCodes.Contains(response.StatusCode);
}
}
}
return false;
}

Use HttpWebRequest rather than WebClient. HttpWebRequest class has a timeout property.

You can try this code:
System.Net.WebRequest r = System.Net.WebRequest.Create("http://www.google.com");
r.Timeout = 3000;
System.Net.WebProxy proxy = new System.Net.WebProxy("<proxy address>");
System.Net.NetworkCredential credentials = new System.Net.NetworkCredential();
credentials.Domain = "<domain>";
credentials.UserName = "<login>";
credentials.Password = "<pass>";
proxy.Credentials = credentials;
r.Proxy = proxy;
try
{
System.Net.WebResponse rsp = r.GetResponse();
}
catch (Exception)
{
MessageBox.Show("Is not avaliable");
return;
}
MessageBox.Show("Avaliable!");

static bool isOnline (string URL)
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(URL);
request.Timeout = 3000;
try
{
WebResponse resp = request.GetResponse();
}
catch (WebException e)
{
if (((HttpWebResponse)e.Response).StatusCode == HttpStatusCode.NotFound)
{
return false;
}
}
return true;
}

can I check if a file exists at a URL?

I know I can locally, on my filesystem, check if a file exists:
if(File.Exists(path))
Can I check at a particular remote URL?

If you're attempting to verify the existence of a web resource, I would recommend using the HttpWebRequest class. This will allow you to send a HEAD request to the URL in question. Only the response headers will be returned, even if the resource exists.
var url = "http://www.domain.com/image.png";
HttpWebResponse response = null;
var request = (HttpWebRequest)WebRequest.Create(url);
request.Method = "HEAD";
try
{
response = (HttpWebResponse)request.GetResponse();
}
catch (WebException ex)
{
/* A WebException will be thrown if the status of the response is not `200 OK` */
}
finally
{
// Don't forget to close your response.
if (response != null)
{
response.Close();
}
}
Of course, if you want to download the resource if it exists it would most likely be more efficient to send a GET request instead (by not setting the Method property to "HEAD", or by using the WebClient class).

If you want to just copy & paste Justin's code and get a method to use, here's how I've implemented it:
using System.Net;
public class MyClass {
static public bool URLExists (string url) {
bool result = false;
WebRequest webRequest = WebRequest.Create(url);
webRequest.Timeout = 1200; // miliseconds
webRequest.Method = "HEAD";
HttpWebResponse response = null;
try {
response = (HttpWebResponse)webRequest.GetResponse();
result = true;
} catch (WebException webException) {
Debug.Log(url +" doesn't exist: "+ webException.Message);
} finally {
if (response != null) {
response.Close();
}
}
return result;
}
}
I'll keep his observation:
If you want to download the resource, and it exists, it would be more efficient to send a GET request instead by not setting the Method property to "HEAD" or by using the WebClient class.

Below is a simplified version of the code:
public bool URLExists(string url)
{
bool result = true;
WebRequest webRequest = WebRequest.Create(url);
webRequest.Timeout = 1200; // miliseconds
webRequest.Method = "HEAD";
try
{
webRequest.GetResponse();
}
catch
{
result = false;
}
return result;
}

If you are using a unc path or a mapped drive, this will work fine.
If you are using a web address (http, ftp etc) you are better off using WebClient - you will get a WebException if it doesn't exist.

public static bool UrlExists(string file)
{
bool exists = false;
HttpWebResponse response = null;
var request = (HttpWebRequest)WebRequest.Create(file);
request.Method = "HEAD";
request.Timeout = 5000; // milliseconds
request.AllowAutoRedirect = false;
try
{
response = (HttpWebResponse)request.GetResponse();
exists = response.StatusCode == HttpStatusCode.OK;
}
catch
{
exists = false;
}
finally
{
// close your response.
if (response != null)
response.Close();
}
return exists;
}

I had the same problem to solve in asp.net core, I've solved with HttpClient
private async Task<bool> isFileExist(string url)
{
using (HttpClient client = new HttpClient())
{
var restponse = await client.GetAsync(url);
return restponse.StatusCode == System.Net.HttpStatusCode.OK;
}
}

My version:
public bool IsUrlExist(string url, int timeOutMs = 1000)
{
WebRequest webRequest = WebRequest.Create(url);
webRequest.Method = "HEAD";
webRequest.Timeout = timeOutMs;
try
{
var response = webRequest.GetResponse();
/* response is `200 OK` */
response.Close();
}
catch
{
/* Any other response */
return false;
}
return true;
}

WebRequest will waiting long time(ignore the timeout user set) because not set proxy, so I change to use RestSharp to do this.
var client = new RestClient(url);
var request = new RestRequest(Method.HEAD);
request.Timeout = 5000;
var response = client.Execute(request);
result = response.StatusCode == HttpStatusCode.OK;

Thanks for all answers.
And I would like to add my implementation which includes default state when we get errors, for specific cases like mine.
private bool HTTP_URLExists(String vstrURL, bool vResErrorDefault = false, int vTimeOut = 1200)
{
bool vResult = false;
WebRequest webRequest = WebRequest.Create(vstrURL);
webRequest.Timeout = vTimeOut; // miliseconds
webRequest.Method = "HEAD";
HttpWebResponse response = null;
try
{
response = (HttpWebResponse)webRequest.GetResponse();
if (response.StatusCode == HttpStatusCode.OK) vResult = true;
else if (response.StatusCode == HttpStatusCode.NotFound) vResult = false;
else vResult = vResErrorDefault;
}
catch (WebException ex)
{
if (ex.Status == WebExceptionStatus.ProtocolError && ex.Response != null)
{
var resp01 = (HttpWebResponse)ex.Response;
if (resp01.StatusCode == HttpStatusCode.NotFound)
{
vResult = false;
}
else
{
vResult = vResErrorDefault;
}
}
else
{
vResult = vResErrorDefault;
}
}
finally
{
// Don't forget to close your response.
if (response != null)
{
response.Close();
}
}
return vResult;
}

Anoter version with define timeout :
public bool URLExists(string url,int timeout = 5000)
{
...
webRequest.Timeout = timeout; // miliseconds
...
}

This works for me:
bool HaveFile(string url)
{
try
{
using (WebClient webClient = new WebClient())
{
webClient.DownloadString(url);
}
return true;
}
catch (Exception)
{
return false;
}
}