I have some fairly simple code that uploads a photo or video to an endpoint (using HTTP PUT or POST). Every so often I see connection closed exceptions thrown, and in reality the photo/video was uploaded just fine, it's calling GetResponse where the exception occurs.
One thing I've noticed is that GetResponse can take an awful long time to process. Often longer than the actual upload time of the photo to the server. My code writes to the web server using RequestStream.Write.
I did a little test and uploaded about 40 photos/videos to the server that range in size from 1MB to 85MB and the time for GetResponse to return was anywhere from 3 to 40 seconds.
My question is, is this normal? Is this just a matter of how long the server I am uploading these files to is taking to process my request and respond? In looking at Fidder HTTP traces it seems to be the case.
FYI, my uploads are HTTP 1.0, Timeout values set to Infinite (both Timeout and ReadWriteTimeout)
If the server is genuinely taking a long time to return any data (as shown in Fiddler) then that's the cause of it. Uploading an 85MB attachment would take a long time to start with, and then the server has to process it. You can't do a lot about that - other than to use an asynchronous method if you're able to get on with more work before the call returns.
It's not entirely clear what Fiddler's showing you though - is it showing a long time before the server sends the response? If so, there's not much you can do. I'm surprised that the connection is being closed on you, admittedly. If, however, you're not seeing your data being written to the server for a while, that's a different matter.
Are you disposing the response returned? If not, you may have connections which are being kept alive. This shouldn't be a problem if it's explicitly HTTP 1.0, but it's the most common cause of "hanging" web calls in my experience.
Basically, if you don't dispose of a WebResponse it will usually (at least with HTTP 1.1 and keepalive) hold on to the connection. There's a limit to the number of connections which can be open to a single host, so you could end up waiting until an earlier response is finalized before the next one can proceed.
If this is the problem, a simple using statement is the answer:
using (WebResponse response = request.GetResponse())
{
...
}
Yes, the response time may be a lot longer than just the upload time. After the request has been sent to the server it has to be processed and a response has to be returned. There may be some time before the request is processed, and then the file typically is going to be saved somewhere. After that the server will create the response page that is sent back.
IIS handles only one request at a time from each user, so if you start another upload before the first one is completed, it will wait until the first one completes before it even starts to process the next.
Related
I have a Dotnet application (running on a Windows PC) which communicates with a Linux box via OPC UA. The use case here is to make ~40 read requests to the server in serial. Once these 40 read calls are complete, the next cycle of 40 read calls begins. Each read call returns a response from the server carrying a payload of ~16KB which is fragmented and delivered to the client. For most requests, the server finishes delivering the complete response within 5ms. However for some requests it takes ~300 ms to complete.
In scenarios where this delay exists, I can see the following pattern of re-transmissions.
[71612] A new Read request is sent to the server.
[71613-71630] The response is delivered to the client.
[71631] A new Read request is sent to the server.
[71632] A TCP Spurious Retransmission occurs from the server for packet [71844] with Seq No. 61624844
[71633] Client sends a DUP ACK for the packet.
[71634] Client does a TCP Retransmission for the read request in [71846] after 288ms
This delay adds up and causes some 5-6 seconds of delay for a complete cycle of 40 requests to complete. I want to figure out what is causing these retransmissions (hence delays) and what can possibly be done to-
Reduce the frequency of retransmissions.
Reduce the 300ms delay from the client side to quickly retransmit the obstructed read request.
I have tried disabling the Nagle algorithm on the server to possibly improve performance but it did not have any effect. Also, when reducing the response size by half (8KB), the retransmissions are rare and hence the delay is minute as well. But reducing the response is not a valid solution in our use case.
The connection to the Linux box is through a switch, however while directly connecting to it point-point, there is marginal reduction in the delay.
I can share relevant code but I think this issue is likely with the TCP stack (or at least, some configuration that should be enabled?) hence it would make little difference.
I'm trying to load various sites to get some information in C# but there are a quite a few sites (not all) where the first request is taking 30-60 seconds.
As an example;
var url = "https://www.coolermaster.com/catalog/cases/mid-tower/masterbox-k501l";
using var wc = new WebClient();
Console.WriteLine(wc.DownloadString(url));
This will consistently take 30-60 seconds every time I first run the app, fiddler says the connection stays in "CONNECT" state but then eventually returns the page content.
I've tried using HttpClient (and disabling proxy via HttpClientHandler), using HttpClientFactory, using WebRequest.Create but no luck.
If I use any command line tools (cURL, wget) it returns instantly, also works instantly doing requests from other languages (JS/Ruby/PHP/Python), but no luck in .NET, even if I send the exact same headers as the other tools send. Tried on various computers, different internet connections, vps, etc.
None of these sites block bots, it's explicitly allowed in robots.txt.
Am I hitting some kind of bug, or is there any option I'm missing? It seems to be related to a slow SSL handshake but I'm not sure?
I have a HttpListener that listen for any file requests on the localhost (i.e 192.168.0.10/Foobar.ext) on the specified port (in this case specifically it is a HLS stream using .m3u8 header files and .ts video files. But the system should work for any type of file).
IAsyncResult result = listener.BeginGetContext(new AsyncCallback(HttpRequestListenerCallback), listener);
result.AsyncWaitHandle.WaitOne(1);
When a request is made the callback creates a HttpListenerContext for the requested file (and that file only), and extracts the file name like this:
HttpListenerContext context = listener.EndGetContext(result);
string fileName = context.Request.Url.AbsolutePath.Substring(1);
the context gets added to a dictionary called httpContexts and linked to an int commandSequenceNumber to keep track of requests.
If the Filename is valid a requests gets send to the server to download the file. The file gets downloaded and gets put into a byte array called totalBufferdata. up to here everything works perfectly.
Now I want to write the byte data of the requested (video)file back to the response (HttpListenerContext.Response) of the context on which the file was requested
To do this i use the following piece of code (this happens after the file has been completely downloaded):
HttpListenerResponse response = httpContexts[commandSequenceNumber].Response; //Get the appropriate context from the dictionary
response.ContentLength64 = totalBufferdata.Count;//set the length response body
Stream responseStream = response.OutputStream; //The stream to which the repsonse needs to be written
try
{
currentContext.Response.OutputStream.Write(dataNoResponseHeader, 0, dataNoResponseHeader.Length);
}
catch (IOException e)
{
Debug.LogError("Exception in writing to response::" + e);
Debug.LogError("innerException:: " + e.InnerException);
}
currentContext.Close();//close the request and response stream
This sends the response back over the context of the request (I.e 192.168.0.10/Foobar.ext, over the same port.)
Now this works fine, aslong as there is a fast, reliable internet connection. When the internet connnection is slow, or inconsistent I start getting the exception:
System.IO.IOException: Unable to write data to the transport connection: The socket has been shut down. ---> System.Net.Sockets.SocketException: The socket has been shut down
With the inner exception being:
System.Net.Sockets.SocketException (0x80004005): The socket has been shut down
I've looked up what the HResult of 0x80004005 corrolated to on msdn, but that is just "E_FAIL Unspecified failure", so no luck there.
I've been unable to figure out why it would throw an expection of the socket closing down (and why it happens on the localhost part, but only when connectivity is bad). I make sure all the data needed is in the totalBufferData, so a low internet speed shouldn't influence this, as all data is already downloaded before i write it to the response. I made sure i do not close the context prematurely anywhere in my code either.
So far i've been unsuccesful in finding a way to get to the underlying socket of HttpListener. I also tried casting response.OutputStream to a NetworkStream, and get the socket from the NetworkStream, but that cast is not valid (Which confused me as they're both IO streams?). Thinking it may be a closing issue i also tried
using(Stream testStream = response.OutputStream)
{
testStream.Write(totalBufferdata.ToArray(), 0, totalBufferdata.Count);
}
I feel like this issue is related to a timeout somewhere. But the Listener doesn't hit the default time-out time. According to MSDN all default time-outs should be 2 minutes. To which I get nowhere close. And i think that the exception returned in case of a time-out should be ObjectDisposedException Write(Byte[], Int32, Int32) was called after the stream was closed. as I would imagine timing out would dispose of the connection, instead of crash it? Is that a misunderstanding?
The requests and reponses are done on seperate threads. But a request that gets done while a response is still in progress gets queued up, and waits for the response to finish before it starts a new response.
Is there a way to get and debug the underlying socket to find out why it closes/crashes? Or maybe an alternative approach to requesting a file over localhost, and responding to this that doesn't use HttpListener?
Some additional information: It is for a Unity application (Unity 2019.1) using scripting runtime version .Net 4.x equivalent, with .Net 4.x api compatibility level and IL2CPP scripting backend. But neither of the classes handling the request or response inherit from monobevahiour (which isn't even possible on threads in unity). Building for Android.
The bounty has ended, but I will open a new one fpr whoever has some valuable information!
See this blog posts that show how to stream a video file using aspnet webapi with the PushStreamContent.
https://www.strathweb.com/2013/01/asynchronously-streaming-video-with-asp-net-web-api/
https://www.c-sharpcorner.com/article/asynchronous-videos-live-streaming-with-asp-net-web-apis-2-0/
This sounds like a problem that I have run into before, with the information given that the internet is slow and or unreliable, I would suggest possibly looking to using a UDP socket instead of TCP as they don't throw exceptions when the connection is cut briefly, or if small amounts of data are lost during transmission, see here. The api is very similar, see here. It would probably be a bit cumbersome to reimplement, but I think it'll solve your problem.
My other insight would be that you're try catch block is specifying that it only accepts IOExceptions, even though it's catching a SocketException, most of the time I just use the generic Exception class to avoid trying to determine which exceptions will be thrown from where.
Just change:
catch (IOException e)
to
catch (Exception e)
Both IOException and SocketException inherit from the Exception class, so the rest of the code remains unmodified. This should give you some more problem specific information hopefully.
You may have some problems there.
First would be a TimeOut situation. Is possible that, because you're experiencing some problems on the internet, that the time to between request and response si bigger than the specified (i believe if you don't, it's set by default to 60 seconds).
Another thing would be that the file size may be to big to write completly at one single package response. But that would happen in any request, not only on the "bad" internet connnection moments.
It's also possible that, because the internet connection is unstable, your "server" detects that the "client" disconected (even briefly), therefore closed the socket.
I am calling an API using these commands:
byte[] messageBytes = System.Text.Encoding.UTF8.GetBytes(message);
var content = new ByteArrayContent(messageBytes);
content.Headers.ContentType = new System.Net.Http.Headers.MediaTypeHeaderValue("application/json");
HttpResponseMessage response = client.PostAsync(ApiUrl, content).Result;
However the code stops executing at the PostAsync line. I put a breakpoint on the next line but it is never reached. It does not throw an error immediately, but a few minutes later it throws an error like:
System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
I presume this means the API is down. What can I do to make PostAsync spit back an error immediately even if the API is down so that I can handle the error and inform the user?
Thank you.
Broadly speaking, what you're asking is "How can I check if an API is available?", and the answer to this depends how low level you want to get, and what you want to do for each level of unavailability:
Is there internet connectivity? Is it worth probing this locally first (as it's relatively quick to check)?
Is the server address correct? If it's wrong it doesn't matter how long you wait. Can the user configure this?
Is the address correct but the server is unable or unwilling to respond? What then?
If you're willing to lump them all into a single "can't contact server in a reasonable amount of time" bucket, there are a few approaches:
Decrease timeouts (beware)
In the case you gave, it sounds like your request is simply timing out: the address or port is wrong, the server is under immense load and can't respond in a timely fashion, you're attempting to contact a non-SSL endpoint using SSL or vice-versa, etc. In any of these cases, you can't know if the request has timed out, until it actually times out. One thing you can do is reduce the HttpClient request timeout. Beware: going too low will cause slow connections to time out on users, which is a worse problem than the one you have.
Pre-check
You could, either before each call, periodically, or at some point early in the client initialisation, do a quick probe of the API to see if it's responsive. This can be spun off into either an async or background task while the UI is being built, etc. This gives you more time to wait for a response, and as an added bonus if the API is responding slowly you can notify your users of this so they know not to expect immediate responses to their clicks. This will improve user experience. If the pre-check fails, you could show an error and advise the user to either check connectivity, check server address (if it's configurable), retry, etc.
Use a CancellationToken
You could pass a CancellationToken into PostAsync with a suitable timeout set, which also allows you to let the user cancel the request if they want to. Read up on CancellationToken for more information.
EDIT: as Alex pointed out, this line is not usually how you deal with async tasks:
HttpResponseMessage response = client.PostAsync(ApiUrl, content).Result;
Change this instead to:
HttpResponseMessage response = await client.PostAsync(ApiUrl, content);
Of course the calling method will then also need to be marked as async, and so on ("It's asyncs, all the way up"), but this is a good thing - it means that your code is not blocking a thread while it waits for a response from the server.
Have a read here for some good material.
Hope that helps
I'm running a small WCF client application that connects to an IIS server every few minutes to download data. There are about 500 of these clients for 2 or 3 servers, and my basic code is something like this:
Client connection = null;
try
{
connection = new Client();
List<TPointer> objects = connection.GetList();
// Some work on List<T>
foreach (TPointer pointer in objects)
{
T data = GetDataFromStream(pointer, connection);
// Some additional processing on T
}
connection.SendMoreData();
// More work
}
catch (...)
{
// Exception handling for various exceptions
}
finally
{
// Handle Close() or Abort()
if (connection != null)
connection.Close();
}
When I simulate running all the clients at once for large amounts of TPointers, I start encountering the following error:
System.TimeoutException: The request channel timed out while waiting for a reply after 00:01:00.
That seems like one of those errors that can occur for any number of reasons. For all I know the server could just be swamped, or I could be requesting too large/too many objects and it's taking too long to download (a whole minute though?). Increasing the timeout is an option, but I'd like to understand the actual problem instead of fixing the symptom.
Given I have no control over the server, how can I streamline my client?
I'm not actually sure what the "request channel" mentioned in the timeout refers to. Does the timeout start ticking from when I create new Client() until I call Client.Close()? Or does each specific request I'm sending to the server (e.g. GetList or GetData) get another minute? Is it worth my while to close Client() in between each call to the server? (I'm hoping not... that would be ugly)
Would it be helpful to chunk up the amount of data I'm receiving? The GetList() call can be quite large (running into the thousands). I could try obtaining a few objects at a time and jobbing off the post-processing for later...
Edit:
Since a few people mentioned streaming:
The Client binding uses TransferMode.StreamedResponse.
GetDataFromStream() uses a Stream derived from TPointer, and SendMoreData()'s payload size is more or less negligible.
Only GetList() actually returns a non-stream object, but I'm unclear as to whether or not that affects the method of transfer.
Or does each specific request I'm sending to the server (e.g. GetList or GetData) get another minute?
The timeout property applies to each and every operation that you're doing. It's reset. If your timeout is one minute, then it starts the moment you invoke that method.
What I'd do is implement a retry policy and use an async version of the client's method and use a CancellationToken or call Abort() on your client when it's taking too long. Alternatively, you can increment or set your timeouts on the InnerChannel on the operation timout.
client.InnerChannel.OperationTimeout = TimeSpan.FromMinutes(10);
You can use that during your operation and in your retry policy you can abort entirely and reset your timeout after your retries have failed or succeeded.
Alternatively, you can try to stream your results and see if you can operate individually on them, but I don't know if keeping that connection open will trip the timeout. You'll have to hold off on operating on your collection until you have everything.
Also, set TransferMode = TransferMode.StreamedResponse in your binding.
I believe the timeout you are hitting is time to first response. In your scenario here first response is the whole response since you are returning the list, more data more time. You might want to consider streaming the data instead of returning a full list.
I suggest to modify both your web.config file (wcf side) and also app.config (client side), adding binding section like this (i.e. timeout of 25 minutes in stead of 1 minute which is default value):
<bindings>
<wsHttpBinding>
<binding name="WSHttpBinding_IYourService"
openTimeout="00:25:00"
closeTimeout="00:25:00"
sendTimeout="00:25:00"
receiveTimeout="00:25:00">
</binding>
</wsHttpBinding>
</bindings>
Given I have no control over the server, how can I streamline my client?
Basically you can not do this when you only have control over the client. It seems like the operations return no Stream (unless the pointers are types which derive from Stream).
If you want to know more about how to generally achieve streaming just read up on this MSDN article.
Everything you can do on the client is scratching on the surface of the problem. Like #The Anathema proposed in his answer you can create a retry logic and/or set the timeout to a higher value. But to eradicate the root of the problem you'd need to investigate the source of the service itself so that it can handle a higher amount of requests. Or have instances of the service running on multiple servers with a load balancer in front.
I ended up going with a combination of the answers here, so I'll just post an answer. I chunked GetList() to a certain size to avoid keeping the connection open so long (it also had a positive effect on the code in general, since I was keeping less in memory temporarily.) I already have a retry policy in place, but will also plan on messing with the timeout, as The Anathema and a couple others suggested.