I am writing a program to crawl Web pages using C# HTTTWebRequest. Since I need to monitor the updates of a specific URL, I write the code as follows. However, I noticed that I could get responses for two times only. After that, it got no responses. Even though I lengthen the period to one minute, I still got the same problem. I don't understand why. Could you give me a hand? Thanks so much!
GreatFree
while (true)
{
WebRequest http = HttpWebRequest.Create("http://www.sina.com.cn");
HttpWebResponse response = (HttpWebResponse)http.GetResponse();
Console.WriteLine(response.LastModified);
Thread.Sleep(5000);
}
imaximchuk's answer is correct, but it doesn't explain why.
HTTP connections are pooled - one connection will be reused for a new request if it's available - and returning the connection to the pool occurs when you close or dispose the web response. In your case, you're not closing or disposing the response, so it's not being returned to the pool. There's a maximum number of connections to any particular host, which is configurable but usually shouldn't be changed. The default is two connections - which is why you're seeing two working responses and then a timeout.
I believe there's a finalizer somewhere which will release the connection when it notices the relevant object is eligible for garbage collection, but that's non-deterministic.
Basically: always close a web response, ideal using a using statement.
while (true)
{
WebRequest http = WebRequest.Create("http://www.sina.com.cn");
using (HttpWebResponse response = (HttpWebResponse)http.GetResponse())
{
Console.WriteLine(response.LastModified);
}
Thread.Sleep(5000);
}
You should call response.Close() when you have finishied working with respose or use using(HttpWebResponse response ... to close it automatically
Related
I have number of APIs which is not being hosted by me, so I have no control of the API itself. What I'm trying to achieve is to check whether the APIs is online or not.I already tried several way:
Sent a HTTP request to the API endpoint with HEAD method
Sent a blank HttpWebRequest
HttpWebRequest request = WebRequest.Create(url) as HttpWebRequest;
using (HttpWebResponse response = request.GetResponse() as HttpWebResponse)
{
if (response.StatusCode == HttpStatusCode.NotFound)
return false;
return true;
}
Ping the server
But somehow my end result is not accurate enough. It shows offline but when I manually try to invoke the API, it seems okay. Anybody got a solution?
What about using an API Health Check service?
Some links that can help you:
https://www.runscope.com/
https://www.getpostman.com/docs/postman/monitors/intro_monitors
https://blog.runscope.com/posts/build-a-health-check-api-for-services-underneath-your-api
https://nordicapis.com/monitor-the-status-of-apis-with-these-4-tools/
https://apigee.com/about/blog/technology/api-health-free-tool-monitoring-apis-public-beta-now
The first option should be good as long as the server can respond to a HEAD request. If it doesn't there should be some safe GET endpoint which can be used instead.
It also makes sense to time the health check so that if the API server is not alive, it won't freeze your app.
You can use Postman Monitors to achieve this. Postman offers 1000 free monitor runs on the free account and notifications via email on failures. You can set it up to run on a schedule by specifying the hour/day/week that the monitor should run at.
Simply create a postman collection: https://learning.getpostman.com/docs/postman/collections/creating_collections/
Add your HTTP Health check request in the collection.
Create a monitor on the collection: https://learning.getpostman.com/docs/postman/monitors/setting_up_monitor/
And set up the frequency that it should run with.
You can also manually trigger monitors via the Postman API: https://docs.api.getpostman.com
I have a small program that keep posting some data (4MB) to remote server. Each time when an internal 4MB buffer is full, I create a new HttpWebRequest, call BeginGetRequestStream/EndGetRequestStream, write data to stream, close it, and done. Because I don't care about response, so I didn't call GetResponse/BeginGetResponse/EndGetResponse.
My question is, since HttpWebRequest doesn't have a Close method, is it safe to leave current HttpWebRequest instance behind? If I didn't do anything to that HttpWebRequest, will the connection be automatically closed? If answer is no, how to explicitly close current HTTP connection? Thanks.
You shouldn't have to cleanup HttpWebRequest but you do need to close the stream coming back from GetRequestStream or the response from GetResponse. I know you said you aren't using any of the response methods, but you must be using one of them to actually make the request.
After you are finished with a WebResponse object, you must close it by calling the Close method. Alternatively, if you have gotten the response stream from the response object, you can close the stream by calling the Stream.Close method. If you do not close the response or the stream, your application can run out of connections to the server and become unable to process additional requests.[1]
[1] https://msdn.microsoft.com/en-us/library/debx8sh9.aspx
We are using an HttpWebRequest in C# to get data from an internet resource in our Azure Web App. The problem is that Azure has a limitation on how long it keeps the connection alive (around 240 seconds). Due to the nature of our application, the response will sometimes take longer than 240 seconds. When this happens, the webpage will go white, and the "View Source" will show zero source code (which has made this issue difficult to debug).
Here's some sample code to illustrate:
webRequest = WebRequest.Create(PAGE_URL) as HttpWebRequest;
webRequest.Method = "POST";
webRequest.ContentType = "application/x-www-form-urlencoded";
webRequest.CookieContainer = cookies;
webRequest.Timeout = Timeout.Infinite;
webRequest.KeepAlive = true;
StreamWriter requestWriter2 = new
StreamWriter(webRequest.GetRequestStream());
requestWriter2.Write(postString);
requestWriter2.Close();
WebResponse response = webRequest.GetResponse();
Stream stream = response.GetResponseStream();
Adding webRequest.Timeout and webRequest.KeepAlive did not solve the issue.
jbq on this thread mentioned he had a workaround by sending a "newline character every 5 seconds", however did not explain how to accomplish this exactly. He was answering a question about an Azure VM, but I think an Azure Web App would have similar behaviors with respect to what I believe are the load balancers that are responsible for the timeout.
The Question:
How can I send one HttpWebRequest, and then send another HttpWebRequest while the previous one is running with a blank line to maintain the connection and prevent the Azure load balancer(?) from timing out the azure application? Would a new session variable need to be used? Perhaps an asynchronous method? Do I need to send the "pinging" request before the main request? If so, how would this look in implementation? Or is it something else entirely? Please provide some source code as an example :)
Note: you do not need to use an HttpWebRequest to replicate this issue. Attach a debugger from within Visual Studio to a live Azure Web App. Place a breakpoint within Visual Studio at any piece of code. When that breakpoint is hit, after roughly 4 minutes you'll see the page in your browser stop loading and go white with an empty source. So, it's not specifically related to HttpWebRequest, but that is an operation that would typically cause this sort of issue since some responses take longer than 4 minutes.
*EDIT: I think what I am looking for is an implementation of Asynchronous methods. I will update this post as I find a satisfactory implementation.
If you are making an HttpWebRequest to an Azure Website then you use ServicePointManager.SetTcpKeepAlive on your client code which uses HttpWebRequest.
https://msdn.microsoft.com/en-us/library/system.net.servicepointmanager.settcpkeepalive(v=vs.110).aspx
The 4 minute timeout that you are talking about is an IDLE timeout over the TCP layer and setting this will ensure that your client (who is using HttpWebRequest) sends ACK packet over TCP so that the connection doesn't get idle.
If your web application is making a HttpWebRequest to some other service, you can still use this function but that will just ensure that the Idle timeout is not hit when calling the remote service. Your actual HTTP request to the Azure Webapp may still hit the 4 minute time and if the client to your Azure web app is not HttpWebRequest, then again the 4 minute idle timeout will bite you...
The best thing to do here is to change the code a bit to implement a JOB Model kind of pattern where-in you make a server call which returns a JOBID. The client then queries the server using this JOBID in a polling fashion and when the job completes on the server the status of this JOBID can be set to COMPLETED in which case the client can then retrieve the data. You can use Webjobs in Azure Webapps to achieve something like this.
Hope this helps...
I'm working on developing a mobile app that is centered around uploading multiple photos to a web api. I'm using Xamarin.Forms and System.Net.Http.HttpClient, and Clumsy to simulate poor network conditions (lag, dropped packets, out-of-order packets). The app was originally written with Titanium, and worked fine for most users, but some users on poor mobile networks were getting frequent errors. Going forward we are porting to Xamarin and trying to accommodate users with poor connectivity.
using (var httpClient = CreateClient())
{
httpClient.Timeout = TimeSpan.FromMinutes(5);
using (var formData = new MultipartFormDataContent())
{
// add required fields via formData.Add() ...
var httpContent = new ByteArrayContent(imageData);
formData.Add(httpContent, "file", Guid.NewGuid() + ".jpg");
try
{
var response = await httpClient.PostAsync("fileupload", formData).ConfigureAwait(false);
if (response.IsSuccessStatusCode)
{
responseObject = await ResponseMessageToResponseModel(response).ConfigureAwait(false);
}
}
catch (HttpRequestException ex)
{
Debug.WriteLine("HttpRequestException");
}
catch (TaskCanceledException ex)
{
Debug.WriteLine("TaskCanceledException");
}
}
}
What I'm finding is that everything works as expected under normal conditions, however; when enabling Clumsy with "lag, drop, out-of-order" and attempting the upload the PostAsync() never completes and eventually times out with TaskCanceledException. The odd thing is that the file ends up on the server.. so the POST data apparently made it through okay.
I'm guessing that packets dropped in the response from the server means the HttpClient never receives a proper response and continues to wait for one until it times out.
To get to the point, I'm wondering if anyone has any ideas on how to make this process as bullet-proof as possible. Just catching the timeout and trying again doesn't work very well if the file made it through the first time. Any thoughts?
Also, any info on how HttpClient handles dropped/out-of-order packets so I can better understand what's happening would be great as well.
One thing with HttpClient I banged my head at a while ago was special(read uncommon) handling of POST requests.
When sending POST request it first sends the headers (including ContentLenght and special Expect: 100-continue header) to the server but without the body. And waits for server to respond with status code 100 if request is acceptable. After that it starts sending the body.
Additional info here:
MSDN Page for ServicePointManager.Expect100Continue
MSDN blog post with some details
In my case the problem was that this part of the protocol wasn't handled particulary well by the backend service (Play framework) I was talking to when request size was too large for it to handle. It was not returning any error. And request simply timed out. So disabling it with
ServicePointManager.Expect100Continue = false;
well ahead of sending any request to this host solved the problem for me. At least now it was returning something.
If that doesn't help than the best thing I can recommend is looking with wireshark or something similar at what is going on on the wire. Provided it plays nice with this Clumsy tool you're using. (Thanks for the link by the way. Was looking for something like this myself)
I have an application which sometimes must do some requests to the server to see that those requests are properly blocked. In other words, the expected server answer is 403 Forbidden.
With a piece of code like this:
HttpWebRequest httpRequest = (HttpWebRequest)WebRequest.Create(protectedPage);
httpRequest.Method = WebRequestMethods.Http.Head;
WebResponse response = HttpRequest.GetResponse();
a WebException is thrown on the last line.
When profiling code, this exception in a try/catch block has a performance impact which I want to avoid.
Is there a way to check for server response, expecting a 403, without having to catch an exception (but avoiding using sockets directly)?
Unfortunately, this is a vexing exception in the framework, and there's no way to avoid it (while still using the HttpWebRequest class). But, as High pointed out, this exception handling will take nanoseconds, while the web request itself will take milliseconds (at best), so don't worry about it.
The WebException object has a Response property that contains the response; when you handle the exception, it's important to dispose that WebResponse object (to avoid leaking connections).
What's taking time in the profiling is probably not the exception catching part but the actual response from the web site.
The WebException.Status property will give you the status code for the HTTP response and you can do further processing based on that.