When running the following Java code, I get very accurate and consistent results in determining if the web page I'm testing is up.
protected synchronized boolean checkUrl(HttpURLConnection connection){
boolean error = false;
//HttpURLConnection connection = null;
GregorianCalendar calendar = new GregorianCalendar();
try{
if(connection != null){
connection.connect();
//200 is the expected HTTP_OK response
error = processResponseCode(connection.getResponseCode());
connection.disconnect();
} else{
error = false;
}
}catch(java.net.UnknownHostException uhe){
... }
catch(Exception e){
... }
return error;
}
The closest match to the Java pattern in c# has much higher results of false positives (mostly due to timeouts - which has a default period of 100000ms).
protected bool connectedToUrl = false;
response = null;
HttpWebRequest webreq = (HttpWebRequest)WebRequest.Create(this.getUri());
webreq.Credentials = CredentialCache.DefaultCredentials;
WebResponse res = null;// webreq.GetResponse();
try
{
WebRequest request = WebRequest.Create(this.getUri()) as WebRequest;
request.Credentials = CredentialCache.DefaultCredentials;
if (request != null)
{
// Get response
res = webreq.GetResponse();
connectedToUrl = processResponseCode(res);
}
else
{
logger.Fatal(getFatalMessage());
string error = string.Empty;
}
}
catch (Exception e)
{
throw e;
}
return connectedToUrl;
}
I have tried various patterns in c# to match the effectiveness of the quoted Java code, to no avail.
Any ideas?
I believe this is because you're not closing any of the request objects.
Simply change this:
res = webreq.GetResponse();
connectedToUrl = processResponseCode(res);
to
using (WebResponse res = webreq.GetResponse())
{
connectedToUrl = processResponseCode(res);
}
(Remove the declaration from earlier.)
Until you haven't closed/disposed the response (or it's been finalized), it's holding onto the connection. You can only have a certain number (2 by default, I believe) of connections to any one host at a time, hence the timeouts. When you dispose the response, it allows another request to use the same connection.
Also this:
catch (Exception e)
{
throw e;
}
Does nothing but destroy the stack trace on an exception that's been bubbled upwards. If you have error handling elsewhere in your code I suggest removing the try catch block. Otherwise you should log the exception and move on. Don't just catch it to throw it.
I think you're missing the GregorianCalendar in the C# version :-)
Why do you have two Request Objects in the C# version?
Related
First of all, thanks for the help you can give me.
Okay, first, excuse my little knowledge, I'm learning, so forgive my ignorance.
I have a method that calls a service, and it is assumed that if any response other than the range of 200 is received, it goes to catch. Well, I need, within that catch, to retrieve from the header (or somewhere) of that response the time it took the service to respond.
I can't use timer or things like that (it was what I first thought of).
The thing is, I don't know how to recover this data from the exception.
Thank you very much!
{
try
{
// Reading the http response
using (HttpWebResponse webResponse = httpWebRequest.GetResponse () as HttpWebResponse)
{
///// Call to endpoint
}
}
catch (WebException ex)
{
//// Here I need to retrieve the time it took for the service to respond (with the error)
}
return response;
} ```
Adding an example to what kshkarin mentioned:
var watch = System.Diagnostics.Stopwatch.StartNew();
try
{
var request = WebRequest.Create("http://www.google.com");
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
using (Stream answer = response.GetResponseStream())
{
// do something
watch.Stop();
Console.WriteLine($"Success at {watch.ElapsedMilliseconds}");
}
}
}
catch (WebException e)
{
// If we got here, it was a timeout exception.
watch.Stop();
Console.WriteLine($"Error occurred at {watch.ElapsedMilliseconds} \n {e}");
}
Indeed you must be set timeout in web request and check this with TimeoutException
if you have some scenario you can check this with simple DateTime
set DateTime.Now and get distance time
Hi I was making a crawler for a site. After about 3 hours of crawling, my app stopped on a WebException. below are my code in c#. client is predefined WebClient object that will be disposed every time gameDoc has already been processed. gameDoc is a HtmlDocument object (from HtmlAgilityPack)
while (retrygamedoc)
{
try
{
gameDoc.LoadHtml(client.DownloadString(url)); // this line caused the exception
retrygamedoc = false;
}
catch
{
client.Dispose();
client = new WebClient();
retrygamedoc = true;
Thread.Sleep(500);
}
}
I tried to use code below (to keep the webclient fresh) from this answer
while (retrygamedoc)
{
try
{
using (WebClient client2 = new WebClient())
{
gameDoc.LoadHtml(client2.DownloadString(url)); // this line cause the exception
retrygamedoc = false;
}
}
catch
{
retrygamedoc = true;
Thread.Sleep(500);
}
}
but the result is still the same. Then I use StreamReader and the result stays the same! below are my code using StreamReader.
while (retrygamedoc)
{
try
{
// using native to check the result
HttpWebRequest webreq = (HttpWebRequest)WebRequest.Create(url);
string responsestring = string.Empty;
HttpWebResponse response = (HttpWebResponse)webreq.GetResponse(); // this cause the exception
using (StreamReader reader = new StreamReader(response.GetResponseStream()))
{
responsestring = reader.ReadToEnd();
}
gameDoc.LoadHtml(client.DownloadString(url));
retrygamedoc = false;
}
catch
{
retrygamedoc = true;
Thread.Sleep(500);
}
}
What should I do and check? I am so confused because I got am able to crawl on some pages, on the same site, then in about 1000 reasults, it cause the exception. the message from exception is only The request was aborted: The connection was closed unexpectedly. and the status is ConnectionClosed
PS. the app is a desktop form app.
update :
Now I am skipping the values and turned them to null so that the crawling can goes on. But if the data is really needed, I still have to update the crawling result manually, which is tiring because the result contains thousands of record. Please help me.
example :
it was like you have downloaded like about 1300 data from the website, then the application stopped saying The request was aborted: The connection was closed unexpectedly. while all your internet connection still on and on a good speed.
ConnectionClosed may indicate (and probably does) that the server you're downloading from is closing the connection. Perhaps it is noticing a large amount of requests from your client and is denying you additional service.
Since you can't control server-side shenanigans, I'd recommend you have some sort of logic to retry the download a bit later.
Got this error because it was returned as 404 from the server.
This issue is really bizarre and has eluded my attempts at debugging. It only occurs when running the app on a Surface tablet. It does not occur on an Asus tablet or while running in Visual Studio. In a particular scenario where Airplane mode has been turned on, a WebException is thrown that I am absolutely unable to catch. I'm not even entirely certain what in my code is causing it to happen, because some of my logging is not happening after a certain point in the code for an unknown reason. I can only assume it's caused by an HttpWebRequest, because of the type of exception being thrown, which appears to be coming from an internal .NET component. Here is the only debugging information I'm able to obtain. It's from the Windows Event Viewer:
Application: <myappname.exe>
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.Net.WebException
Stack:
at System.Net.ServicePoint.ConnectSocketCallback(System.IAsyncResult)
at System.Net.LazyAsyncResult.Complete(IntPtr)
at System.Net.ContextAwareResult.Complete(IntPtr)
at System.Net.LazyAsyncResult.ProtectedInvokeCallback(System.Object, IntPtr)
at System.Net.Sockets.Socket.ConnectCallback()
at System.Net.Sockets.Socket.RegisteredWaitCallback(System.Object, Boolean)
at System.Threading._ThreadPoolWaitOrTimerCallback.PerformWaitOrTimerCallback(System.Object, Boolean)
I really wish I had more debugging information to provide, but I've tried everything I can think of with tons of try/catch blocks everywhere and logging after most calls--some of which isn't be executed. Does anyone have any clue as to what could be causing this?
EDIT 1:
Based on traces the exception appears to be thrown somewhere in here. Virtually everything is wrapped in a try/catch block, so I can't see how the WebException could possibly be slipping through.
byte[] bytes = Encoding.UTF8.GetBytes(requestXml.ToString());
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(url);
request.Method = "POST";
request.ContentType = "text/xml";
try
{
IAsyncResult requestResult = (IAsyncResult)request.BeginGetRequestStream((rAsyncResult) =>
{
using (Stream uploadStream = request.EndGetRequestStream(rAsyncResult))
{
try
{
uploadStream.Write(bytes, 0, bytes.Length);
uploadStream.Flush();
}
catch (Exception e)
{
// Exception handling
}
finally
{
uploadStream.Dispose();
}
}
IAsyncResult responseResult = (IAsyncResult)request.BeginGetResponse((resAsyncResult) =>
{
try
{
using (HttpWebResponse response = (HttpWebResponse)request.EndGetResponse(resAsyncResult))
{
try
{
data = ProcessResponse(XmlReader.Create(response.GetResponseStream()));
}
catch (Exception e)
{
// Exception handling
}
finally
{
response.Dispose();
}
}
}
catch (WebException e)
{
// Exception handling
}
}, null);
}, null);
}
catch (Exception e)
{
// Exception handling
}
EDIT 2:
I still have not found an acceptable solution. I'm currently checking the connection type before-hand and not allowing the code to continue if it's not connected to WiFi, Mobile, or Ethernet, but that doesn't catch the condition where it's connected to a network that has no Internet connection. WinRT has no solution to check Internet connectivity, and even the method I'm using is unfriendly to work with (it just passes back a number--http://social.msdn.microsoft.com/Forums/en-US/winappswithcsharp/thread/d8e76732-19d3-47b3-840f-70d87c75ce9f).
Did you try handling the Application.UnhandledException?
Add the event handler to the event in the App class constructor:
public App()
{
// keep the rest of the constructor
this.UnhandledException += OnUnhandledException;
}
In the event handler you can log the exception and mark is handled to prevent the app from closing / crashing:
void OnUnhandledException(object sender, UnhandledExceptionEventArgs e)
{
// log e.Exception details for investigation
e.Handled = true;
}
The issue is probably due to an unhandled exception thrown in your callback; chances are high that the callback is executed asynchronously in a thread different from the one that called the initial request.BeginGetRequestStream and this is why you aren't catching it in the outer try/catch block.
You should be able to overcome this problem by wrapping the entire content of the callback in a try/catch block, that is:
IAsyncResult requestResult = (IAsyncResult)request.BeginGetRequestStream((rAsyncResult) =>
{
try
{
using (Stream uploadStream = request.EndGetRequestStream(rAsyncResult))
{
try
{
uploadStream.Write(bytes, 0, bytes.Length);
uploadStream.Flush();
}
catch (Exception e)
{
// Exception handling
}
finally
{
uploadStream.Dispose();
}
}
IAsyncResult responseResult = (IAsyncResult)request.BeginGetResponse((resAsyncResult) =>
{
try
{
using (HttpWebResponse response = (HttpWebResponse)request.EndGetResponse(resAsyncResult))
{
try
{
data = ProcessResponse(XmlReader.Create(response.GetResponseStream()));
}
catch (Exception e)
{
// Exception handling
}
finally
{
response.Dispose();
}
}
}
catch (WebException e)
{
// Exception handling
}
}, null);
}
catch (Exception ex)
{
// Handle the exception as you wish
}
}, null);
Like Efran Cobisi said EndGetRequestStream is probably the function throwing the exception. Also a using statement will dispose the an object even if there is an exception so no need of a try finally to dispose it.
But in any case you should be using the async methods, that will make the code a lot more readable and exception easier to catch. The equivalent of your code will be:
byte[] bytes = Encoding.UTF8.GetBytes(requestXml.ToString());
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(url);
request.Method = "POST";
request.ContentType = "text/xml";
try
{
using (Stream uploadStream = await request.GetRequestStreamAsync())
{
await uploadStream.WriteAsync(bytes, 0, bytes.Length);
await uploadStream.FlushAsync();
}
using (HttpWebResponse response = (HttpWebResponse) await request.GetRequestStreamAsync())
{
data = ProcessResponse(XmlReader.Create(response.GetResponseStream()));
}
}
catch (Exception e)
{
// Exception
}
I have a weird problem and I am not sure where is it coming from.
Started last night when I started my application.
I have the following code:
DateTime startTime = DateTime.Now;
WebRequest req = (WebRequest)HttpWebRequest.Create("http://" + server + url + action);
WebResponse res = req.GetResponse();
try
{
using (StreamReader reader = new StreamReader(res.GetResponseStream()))
{
string response = reader.ReadToEnd();
TestInfo.CheckMsg retMsg;
// individual Test
retMsg = indFunc(req.RequestUri.ToString(), response);
printMessage(retMsg, req, res, startTime);
if (retMsg.SeverityLevel > TestInfo.CheckMsg.Severity.Warning)
return;
// group Test
retMsg = groupFunc(relatedGroup, req.RequestUri.ToString(), response);
printMessage(retMsg, req, res, startTime);
if (retMsg.SeverityLevel > TestInfo.CheckMsg.Severity.Warning)
return;
// common Test
retMsg = commonFunc(req.RequestUri.ToString(), response);
printMessage(retMsg, req, res, startTime);
if (retMsg.SeverityLevel > TestInfo.CheckMsg.Severity.Warning)
return;
reader.Close();
};
}
catch (Exception e)
{
Console.WriteLine("EXCEPTION WAS THROWN!!!!! " + e.Message);
}
finally
{
res.Close();
}
This function runs on several threads.
After several application debugging this code fails to connect claiming UNDERLYING CONNECTION WAS CLOSED.
when trying to browse in my computer (WIN XP) in all browsers to all sites I get either blank page or in chrome I get that the response from the server was empty.
Http only is screwed while other TCP connections work!
So I guess the problem might be in my code (maybe something is not released???)
Let me know what do you think of the code.
thanks
One thing I do see wrong is that you need to move
WebResponse res = req.GetResponse();
Into your try block bc it can throw an exception(MSDN). I would also add a null check to:
if(res != null){
res.Close()
}
There are times that server shutdown, file missing on server and other problem. So, I want to trap or catch the exception thrown by HttpWebRequest when using Dispatcher thread to update content on UI.
The below code is unable to catch error and display in MessageBox.show(). Can anyone show me what I need to do? Thanks
HttpWebRequest webReq;
HttpWebResponse webResp;
public void GetInfo(string Url)
{
webReq = (HttpWebRequest)HttpWebRequest.Create(Url);
try
{
webReq.BeginGetResponse(OnGetBuffer, this);
}
catch (Exception e)
{
}
}
public void OnGetBuffer(IAsyncResult asr)
{
webResp = (HttpWebResponse)webReq.EndGetResponse(asr);
Deployment.Current.Dispatcher.BeginInvoke(() =>
{
Stream streamResult = webResp.GetResponseStream();
try
{
}
catch (Exception)
{
}
});
}
Place a try/catch around the .EndGetResponse() call. I believe this is where the exception is being thrown.
First of all, I hope you don't intend to catch all exceptions and ignore them all. You would be ignoring exceptions that have nothing to do with your network connection failing.
Second, you need to place the try/catch around the code that might throw the exception:
public void OnGetBuffer(IAsyncResult asr)
{
HttpWebResponse webResp;
try
{
webResp = (HttpWebResponse)webReq.EndGetResponse(asr);
}
Catch (WebException ex)
{
// Do something to decide whether to retry, then retry or else
throw; // Re-throw if you're not going to handle the exception
}
Deployment.Current.Dispatcher.BeginInvoke(() =>
{
using (Stream streamResult = webResp.GetResponseStream())
{
// Do something with the stream
}
});
}
try using the WebClient object. Then in the completed event handler the error is returned as e.Error