I am using the code below to post to a web site but I get a 411 error, (
The remote server returned an error: (411) Length Required).
This is the function I am using, I just removed the exception handling. I get a WebException been thrown.
private static async Task<string> PostAsync(string url, string queryString, Dictionary<string, string> headers)
{
var webRequest = (HttpWebRequest)WebRequest.Create(url);
webRequest.ContentType = "application/x-www-form-urlencoded";
webRequest.Method = "POST";
if (headers != null)
{
foreach (var header in headers)
{
webRequest.Headers.Add(header.Key, header.Value);
}
}
if (!string.IsNullOrEmpty(queryString))
{
queryString = BuildQueryString(query);
using (var writer = new StreamWriter(webRequest.GetRequestStream()))
{
writer.Write(queryString);
}
}
//Make the request
try
{
using (
var webResponse = await Task<WebResponse>.Factory.FromAsync(webRequest.BeginGetResponse, webRequest.EndGetResponse, webRequest).ConfigureAwait(false))
{
using (var str = webResponse.GetResponseStream())
{
if (str != null)
{
using (var sr = new StreamReader(str))
{
return sr.ReadToEnd();
}
}
return null;
}
}
}
catch (WebException wex)
{
// handle webexception
}
catch (Exception ex)
{
// handle webexception
}
}
I saw on some site that adding
webRequest.ContentLength = 0;
Would work, but in some cases I get errors that the length is wrong, (so it must be something other than 0).
So my questions are, how do I set the content length properly?
And, am I sending my post requests properly? Is there another way?
Not sure if it matters, but I am using .NET 4.6, (but I can use 4.6.1 if needed).
The length is the length of the data you're writing to the request stream. So in this bit, set the ContentLength:
using (var writer = new StreamWriter(webRequest.GetRequestStream()))
{
writer.Write(queryString);
request.ContentLength = queryString.Length;
}
Note that you might have trouble if queryString contains unicode characters. The ContentLength indicates the number of bytes, but string.Length indicates the number of characters. Since some unicode characters take up more than one byte, there can possibly be a mismatch. There are ways to compensate if need be.
As an alternative, you can use HttpClient instead. I haven't had to set the ContentLength manually when using it.
If you are using POST and you need to use POST then you need this where data represents some data you are posting be it JSON, XML or whatever:
request.ContentLength = data.Length;
Having said that, I think you are probably ok with using GET instead since you are not sending nothing in the body of the request. But in some cases you have no choice but to use POST.
I am writing a tool that allows the user to input a URL, to which the program responds by attempting to show that website's favicon. I have this working for many sites but one site that is giving me trouble is my self-hosted Trac site. It seems that Trac's normal behaviour, until the end user is autenticated, is to show a custom 403 page (Forbidden), inviting the user to log in. Accessing Trac from a web browser, the favicon displays in the browser's tab, even though I'm not logged in (and Firebug, for instance, shows a 403 for the page content). If I view source from the browser, the favicon's location is right there in the source. However, from my application, requesting the Trac website with request.GetResponse() throws a WebException containing a 403, giving me no opportunity to read the response stream that contains the vital information required to find the favicon.
I already have code to download a website's HTML and extract the location of its favicon. What I am stuck with is downloading a site's HTML even when it responds with a 403.
I played with various UserAgent, Accept and AcceptLanguage properties of the HttpWebRequest object but it didn't help. I also tried following any redirects myself as I read somewhere that .NET doesn't do them well. Still no luck.
Here's what I have:
public static MemoryStream DownloadHtml(
string urlParam,
int timeoutMs = DefaultHttpRequestTimeoutMs,
string userAgent = "",
bool silent = false
)
{
MemoryStream result = null;
HttpWebRequest request = null;
HttpWebResponse response = null;
try
{
Func<string, HttpWebRequest> createRequest = (urlForFunc) =>
{
var requestForAction = (HttpWebRequest)HttpWebRequest.Create(urlForFunc);
// This step is now required by Wikipedia (and others?) to prevent periodic or
// even constant 403's (Forbidden).
requestForAction.UserAgent = userAgent;
requestForAction.Accept = "text/html";
requestForAction.AllowAutoRedirect = false;
requestForAction.Timeout = timeoutMs;
return requestForAction;
};
string urlFromResponse = "";
string urlForRequest = "";
do
{
if(response == null)
{
urlForRequest = urlParam;
}
else
{
urlForRequest = urlFromResponse;
response.Close();
}
request = createRequest(urlForRequest);
response = (HttpWebResponse)request.GetResponse();
urlFromResponse = response.Headers[HttpResponseHeader.Location];
}
while(urlFromResponse != null
&& urlFromResponse.Length > 0
&& urlFromResponse != urlForRequest);
using(var stream = response.GetResponseStream())
{
result = new MemoryStream();
stream.CopyTo(result);
}
}
catch(WebException ex)
{
// Things like 404 and, well, all other web-type exceptions.
Debug.WriteLine(ex.Message);
if(ex.InnerException != null) Debug.WriteLine(ex.InnerException.Message);
}
catch(System.Threading.ThreadAbortException)
{
// Let ac.Thread handle some cleanup.
throw;
}
catch(Exception)
{
if(!silent) throw;
}
finally
{
if(response != null) response.Close();
}
return result;
}
The stream content is stored in Exception object.
var resp = new StreamReader(ex.Response.GetResponseStream()).ReadToEnd();
I am playing around with an app using HttpWebRequest to dialog with a web server.
I followed standard instructions I found on the web to build my request function that I tried to make as generic as possible (I try to get a unique method regardless of the method: PUT, POST, DELETE, REPORT, ...)
When I submit a "REPORT" request, I get two access logs on my server:
1) I get response 401 after following line is launched in debugger
reqStream.Write(Encoding.UTF8.GetBytes(body), 0, body.Length);
2) I get response 207 (multi-get, which is what I expect) after passing the line calling Request.GetResponse();
Actually, it seems to be the Request.GetRequestStream() line that is querying the server the first time, but the request is only committed once passing the reqStream.Write(...) line...
Same for PUT and DELETE, the Request.GetRequestStream() again generates a 401 access log on my server whereas the Request.GetResponse(); returns code 204.
I don't understand why for a unique request I have two server access logs, especially one that seems to be doing nothing as it always returns code 401... Could anybody explain what is going on? Is it a flaw in my code or a bad design due to my attempt to get a generic code for multiple methods?
Here is my full code:
public static HttpWebResponse getHttpWebRequest(string url, string usrname, string pwd, string method, string contentType,
string[] headers, string body) {
// Variables.
HttpWebRequest Request;
HttpWebResponse Response;
//
string strSrcURI = url.Trim();
string strBody = body.Trim();
try {
// Create the HttpWebRequest object.
Request = (HttpWebRequest)HttpWebRequest.Create(strSrcURI);
// Add the network credentials to the request.
Request.Credentials = new NetworkCredential(usrname.Trim(), pwd);
// Specify the method.
Request.Method = method.Trim();
// request headers
foreach (string s in headers) {
Request.Headers.Add(s);
}
// Set the content type header.
Request.ContentType = contentType.Trim();
// set the body of the request...
Request.ContentLength = body.Length;
using (Stream reqStream = Request.GetRequestStream()) {
// Write the string to the destination as a text file.
reqStream.Write(Encoding.UTF8.GetBytes(body), 0, body.Length);
reqStream.Close();
}
// Send the method request and get the response from the server.
Response = (HttpWebResponse)Request.GetResponse();
// return the response to be handled by calling method...
return Response;
}
catch (Exception e) {
throw new Exception("Web API error: " + e.Message, e);
}
I'm currently writing a simple app that performs a series of requests to the web server and I've encountered a strange... feature?
I don't need response stream of the request, but only status code. So, for each piece of my data I call my own "Send" method:
public static int Send(string uri)
{
HttpWebRequest request = null;
HttpWebResponse response = null;
try
{
request = (HttpWebRequest)WebRequest.Create(uri);
response = (HttpWebResponse)request.GetResponse();
if (response.StatusCode == HttpStatusCode.OK) return 0;
}
catch (Exception e)
{
if (request != null) request.Abort();
}
return -1;
}
Works fine? Yes, unless I call this function at least twice. Second call of such a function in a row (with the same uri) will ALWAYS result in timeout.
Now, that's odd: if I add request.Abort(); when I return zero (here, when status code is 200) - everything ALWAYS works fine.
So my question is - why? Is it some kind of framework restriction, or maybe the some kind of anti-DOS protection on the particular server (unfortunately, the server is a black box for me)? Or maybe I just don't understand smth in how it all works?
Try to dispose of the web response, you may leak some resources
public static int Send(string uri)
{
HttpWebRequest request = null;
try
{
request = (HttpWebRequest)WebRequest.Create(uri);
using (var response = (HttpWebResponse)request.GetResponse())
{
if (response.StatusCode == HttpStatusCode.OK) return 0;
}
}
catch (Exception e)
{
if (request != null) request.Abort();
}
return -1;
}
There is also a default number of connections (2 I think, but you can configure this) you can make to a domain simultaneously, please see this SO question. You're probably hitting this limit with your unclosed responses.
First of all I'd make a series of changes in order to get to the root of this:
take out that try..catch{} (you're likely swallowing an exception)
return a boolean instead of a number
You should then get your exception information you need.
Also you should be using "HEAD" as your method as you only want the status code:
request.Method = "HEAD";
read the difference here.
I am making a simple program in visual c# 2005 that looks up a stock symbol on Yahoo! Finance, downloads the historical data, and then plots the price history for the specified ticker symbol.
I know the exact URL that I need to acquire the data, and if the user inputs an existing ticker symbol (or at least one with data on Yahoo! Finance) it works perfectly fine. However, I have a run-time error if the user makes up a ticker symbol, as the program tries to pull data from a non-existent web page.
I am using the WebClient class, and using the DownloadString function. I looked through all the other member functions of the WebClient class, but didn't see anything I could use to test a URL.
How can I do this?
Here is another implementation of this solution:
using System.Net;
///
/// Checks the file exists or not.
///
/// The URL of the remote file.
/// True : If the file exits, False if file not exists
private bool RemoteFileExists(string url)
{
try
{
//Creating the HttpWebRequest
HttpWebRequest request = WebRequest.Create(url) as HttpWebRequest;
//Setting the Request method HEAD, you can also use GET too.
request.Method = "HEAD";
//Getting the Web Response.
HttpWebResponse response = request.GetResponse() as HttpWebResponse;
//Returns TRUE if the Status code == 200
response.Close();
return (response.StatusCode == HttpStatusCode.OK);
}
catch
{
//Any exception will returns false.
return false;
}
}
From: http://www.dotnetthoughts.net/2009/10/14/how-to-check-remote-file-exists-using-c/
You could issue a "HEAD" request rather than a "GET"?
So to test a URL without the cost of downloading the content:
// using MyClient from linked post
using(var client = new MyClient()) {
client.HeadOnly = true;
// fine, no content downloaded
string s1 = client.DownloadString("http://google.com");
// throws 404
string s2 = client.DownloadString("http://google.com/silly");
}
You would try/catch around the DownloadString to check for errors; no error? It exists...
With C# 2.0 (VS2005):
private bool headOnly;
public bool HeadOnly {
get {return headOnly;}
set {headOnly = value;}
}
and
using(WebClient client = new MyClient())
{
// code as before
}
These solutions are pretty good, but they are forgetting that there may be other status codes than 200 OK. This is a solution that I've used on production environments for status monitoring and such.
If there is a url redirect or some other condition on the target page, the return will be true using this method. Also, GetResponse() will throw an exception and hence you will not get a StatusCode for it. You need to trap the exception and check for a ProtocolError.
Any 400 or 500 status code will return false. All others return true.
This code is easily modified to suit your needs for specific status codes.
/// <summary>
/// This method will check a url to see that it does not return server or protocol errors
/// </summary>
/// <param name="url">The path to check</param>
/// <returns></returns>
public bool UrlIsValid(string url)
{
try
{
HttpWebRequest request = HttpWebRequest.Create(url) as HttpWebRequest;
request.Timeout = 5000; //set the timeout to 5 seconds to keep the user from waiting too long for the page to load
request.Method = "HEAD"; //Get only the header information -- no need to download any content
using (HttpWebResponse response = request.GetResponse() as HttpWebResponse)
{
int statusCode = (int)response.StatusCode;
if (statusCode >= 100 && statusCode < 400) //Good requests
{
return true;
}
else if (statusCode >= 500 && statusCode <= 510) //Server Errors
{
//log.Warn(String.Format("The remote server has thrown an internal error. Url is not valid: {0}", url));
Debug.WriteLine(String.Format("The remote server has thrown an internal error. Url is not valid: {0}", url));
return false;
}
}
}
catch (WebException ex)
{
if (ex.Status == WebExceptionStatus.ProtocolError) //400 errors
{
return false;
}
else
{
log.Warn(String.Format("Unhandled status [{0}] returned for url: {1}", ex.Status, url), ex);
}
}
catch (Exception ex)
{
log.Error(String.Format("Could not test url {0}.", url), ex);
}
return false;
}
If I understand your question correctly, you could use a small method like this to give you the results of your URL test:
WebRequest webRequest = WebRequest.Create(url);
WebResponse webResponse;
try
{
webResponse = webRequest.GetResponse();
}
catch //If exception thrown then couldn't get response from address
{
return 0;
}
return 1;
You could wrap the above code in a method and use it to perform validation. I hope this answers the question you were asking.
Try this (Make sure you use System.Net):
public bool checkWebsite(string URL) {
try {
WebClient wc = new WebClient();
string HTMLSource = wc.DownloadString(URL);
return true;
}
catch (Exception) {
return false;
}
}
When the checkWebsite() function gets called, it tries to get the source code of
the URL passed into it. If it gets the source code, it returns true. If not,
it returns false.
Code Example:
//The checkWebsite command will return true:
bool websiteExists = this.checkWebsite("https://www.google.com");
//The checkWebsite command will return false:
bool websiteExists = this.checkWebsite("https://www.thisisnotarealwebsite.com/fakepage.html");
I have always found Exceptions are much slower to be handled.
Perhaps a less intensive way would yeild a better, faster, result?
public bool IsValidUri(Uri uri)
{
using (HttpClient Client = new HttpClient())
{
HttpResponseMessage result = Client.GetAsync(uri).Result;
HttpStatusCode StatusCode = result.StatusCode;
switch (StatusCode)
{
case HttpStatusCode.Accepted:
return true;
case HttpStatusCode.OK:
return true;
default:
return false;
}
}
}
Then just use:
IsValidUri(new Uri("http://www.google.com/censorship_algorithm"));
A lot of the answers are older than HttpClient (I think it was introduced in Visual Studio 2013) or without async/await functionality, so I decided to post my own solution:
private static async Task<bool> DoesUrlExists(String url)
{
try
{
using (HttpClient client = new HttpClient())
{
//Do only Head request to avoid download full file
var response = await client.SendAsync(new HttpRequestMessage(HttpMethod.Head, url));
if (response.IsSuccessStatusCode) {
//Url is available is we have a SuccessStatusCode
return true;
}
return false;
}
} catch {
return false;
}
}
I use HttpClient.SendAsync with HttpMethod.Head to make only a head request, and not downlaod the whole file. Like David and Marc already say there is not only http 200 for ok, so I use IsSuccessStatusCode to allow all Sucess Status codes.
WebRequest request = WebRequest.Create("http://www.google.com");
try
{
request.GetResponse();
}
catch //If exception thrown then couldn't get response from address
{
MessageBox.Show("The URL is incorrect");`
}
This solution seems easy to follow:
public static bool isValidURL(string url) {
WebRequest webRequest = WebRequest.Create(url);
WebResponse webResponse;
try
{
webResponse = webRequest.GetResponse();
}
catch //If exception thrown then couldn't get response from address
{
return false ;
}
return true ;
}
Here is another option
public static bool UrlIsValid(string url)
{
bool br = false;
try {
IPHostEntry ipHost = Dns.Resolve(url);
br = true;
}
catch (SocketException se) {
br = false;
}
return br;
}
A lot of other answers are using WebRequest which is now obsolete.
Here is a method that has minimal code and uses currently up-to-date classes and methods.
I have also tested the other most up-voted functions which can produce false positives.
I tested with these URLs, which points to the Visual Studio Community Installer, found on this page.
//Valid URL
https://aka.ms/vs/17/release/vs_community.exe
//Invalid URL, redirects. Produces false positive on other methods.
https://aka.ms/vs/14/release/vs_community.exe
using System.Net;
using System.Net.Http;
//HttpClient is not meant to be created and disposed frequently.
//Declare it staticly in the class to be reused.
static HttpClient client = new HttpClient();
/// <summary>
/// Checks if a remote file at the <paramref name="url"/> exists, and if access is not restricted.
/// </summary>
/// <param name="url">URL to a remote file.</param>
/// <returns>True if the file at the <paramref name="url"/> is able to be downloaded, false if the file does not exist, or if the file is restricted.</returns>
public static bool IsRemoteFileAvailable(string url)
{
//Checking if URI is well formed is optional
Uri uri = new Uri(url);
if (!uri.IsWellFormedOriginalString())
return false;
try
{
using (HttpRequestMessage request = new HttpRequestMessage(HttpMethod.Head, uri))
using (HttpResponseMessage response = client.Send(request))
{
return response.IsSuccessStatusCode && response.Content.Headers.ContentLength > 0;
}
}
catch
{
return false;
}
}
Just note that this will not work with .NET Framework, as HttpClient.Send does not exist.
To get it working on .NET Framework you will need to change client.Send(request) to client.SendAsync(request).Result.
Web servers respond with a HTTP status code indicating the outcome of the request e.g. 200 (sometimes 202) means success, 404 - not found etc (see here). Assuming the server address part of the URL is correct and you are not getting a socket timeout, the exception is most likely telling you the HTTP status code was other than 200. I would suggest checking the class of the exception and seeing if the exception carries the HTTP status code.
IIRC - The call in question throws a WebException or a descendant. Check the class name to see which one and wrap the call in a try block to trap the condition.
i have a more simple way to determine weather a url is valid.
if (Uri.IsWellFormedUriString(uriString, UriKind.RelativeOrAbsolute))
{
//...
}
Following on from the examples already given, I'd say, it's best practice to also wrap the response in a using like this
public bool IsValidUrl(string url)
{
try
{
var request = WebRequest.Create(url);
request.Timeout = 5000;
request.Method = "HEAD";
using (var response = (HttpWebResponse)request.GetResponse())
{
response.Close();
return response.StatusCode == HttpStatusCode.OK;
}
}
catch (Exception exception)
{
return false;
}
}