Consider the situation where some client calls WebMethod and after connection was established the host process crashes and connection becomes half-open. Will ASP.NET WebMethod proxy automatically detect it?
We assume that WebMethod timeout settled to the infinity or some big amount of time. Will connectivity error be detected only on TCP keep-alive packet sending (after 2 hours of idle on Windows by default)?
[WebService(Namespace = "http://www.example.com/TestService/")]
[WebServiceBinding(ConformsTo = WsiProfiles.BasicProfile1_1)]
public class TestService : System.Web.Services.WebService
{
[WebMethod]
public void DoLongWork()
{
// long work here
}
}
Web Reference proxy call:
public void ServiceClient()
{
var serviceProxy = new MyNamespace.TestService();
// serviceProxy.Url = ...
serviceProxy.Timeout = -1; // do not use timeout
// will it wait for 2 hours (when TCP keep-alive will be sent) if connection was lost after handshake?
serviceProxy.DoLongWork();
}
UPDATE.
I've tried multiple scenarios and here the results that I got using Wireshark to see what is going on under the hood.
Stopping hosting Web Server (IIS) through the IIS Manager or killing host process (w3wp.exe):
In both cases client notified via sending TCP RST flag. Seems Windows takes care of closing sockets for crashed/killed processes. In .NET the System.Net.WebException will be thrown:
Unhandled Exception: System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a receive. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
--- End of inner exception stack trace ---
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.PooledStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.Connection.SyncRead(HttpWebRequest request, Boolean userRetrievedStream, Boolean probeRead)
--- End of inner exception stack trace ---
at System.Web.Services.Protocols.WebClientProtocol.GetWebResponse(WebRequest request)
at System.Web.Services.Protocols.HttpWebClientProtocol.GetWebResponse(WebRequest request)
at System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String methodName, Object[] parameters)
at WebMethodAndTCPHalfOpenTest.TestWebReference.TestService.DoLongWork(Int32 sec)
at WebMethodAndTCPHalfOpenTest.Program.Main(String[] args)
Killed the host process after connectivity (to the host) loss, so TCP RST can not reach client:
Results here is very tangling for me, because after 2.5 hours the broken connection (half-open) is not detected and client still thinks that underlying TCP connection is alive (the same situation after 6 hours). Here the output from Sysinternals TCPView:
TCPView v3.01 - TCP/UDP endpoint viewer Copyright (C) 1998-2010
Mark Russinovich and Bryce Cogswell Sysinternals -
www.sysinternals.com
[TCP] WebMethodAndTCPHalfOpenTest.exe
PID: 8180
State: ESTABLISHED
Local: 192.168.1.2
Remote: 5.167.159.81
Windows Version: Microsoft Windows [Version 6.1.7601]
Wireshark output:
I've checked KeepAliveTime registry setting on my system here: HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters and it was missing, so the default 2 hours should be used (?).
Can anybody explain the result described in the second point? Why TCP keepalive was not sent? Is proxy explicitly settled up different keepalive behavior for underlying socket (https://msdn.microsoft.com/en-us/library/windows/desktop/ee470551(v=vs.85).aspx)?
It looks like the keep alive is not garanteed because things might muck with that setting in windows (no the registry value isn't the end all).
According to this:
Windows TCP socket has SO_KEEPALIVE enabled by default?
It looks like you should not rely on this behavior and set a timeout. If you are using a soap request, you could always invoke an async request, and then wait on the request based on a client configurable value, and have a nice logic handling for timeouts.
If the host process crashes the connection will close. It won't wait for the TCP keep-alive to kick. You can easily test this scenario by spinning your WebService and making it crash voluntary.
Related
Background:
We have a common service to download a file to specific location. This service uses WebClient to download a file. This service is used by multiple applications.
.Net Framenwork: 4.5.2
Enabled TLS version on Prod, UAT: TLs1.3, TLS1.2
Issue:
Sometimes We are receiving an exception mentioned below on Prod server only not QA, UAT.
An exception occurred during a WebClient request.
InnerException :
System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
at System.Net.Sockets.Socket.BeginReceive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags, AsyncCallback callback, Object state)
at System.Net.Sockets.NetworkStream.BeginRead(Byte[] buffer, Int32 offset, Int32 size, AsyncCallback callback, Object state)
--- End of inner exception stack trace ---
at System.Net.ConnectStream.EndRead(IAsyncResult asyncResult)
at System.Net.WebClient.DownloadBitsReadCallbackState(DownloadBitsState state, IAsyncResult result)
Query: When looked for this type of issue almost every page on internet is saying it TLS issue, just add a line mention below in code. And few pages are saying don't hardcode the TLS version.
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
I also believe we should not hardcode the tls version.
But I am not getting what is the actual root cause of this issue?
why this issue is intermittent and why this persist on only Prod?
Code:
WebClient wc = new WebClient();
await wc.DownloadFileTaskAsync(new Uri(webendpoint), sharedPath);
return 0;
I have a .Net web service running on a Windows 2012 64-bit application server, IIS 8.
An issue was raised by our customer a number of weeks back and we are finding it very difficult to diagnose where the problem lies!
The customer calls a method on our web service passing an XML data structure. The web method simply inserts that xml data into a SQL Server database table. The return from the web method is a string "OK".
60% of the time the customer reports no issues. But for the other 40%, they are receiving the error...
"The underlying connection was closed: The connection was closed
unexpectedly."
Here's a more detailed trace from the customer's end...
Message: The underlying connection was closed: The connection was
closed unexpectedly. HResult: -2146233079 Source: System StackTrace:
at System.Net.HttpWebRequest.GetResponse() Message: Unable to read
data from the transport connection: The connection was closed.
HResult: -2146232800 Source: System StackTrace: at
System.Net.ConnectStream.InternalWrite(Boolean async, Byte[] buffer,
Int32 offset, Int32 size, AsyncCallback callback, Object state) at
System.Net.ConnectStream.Write(Byte[] buffer, Int32 offset, Int32
size)
Now here's the dilemma. For those failures, we did actually receive the XML data! I have even modified the web service to write to a log file just before the method returns the "OK" value. All looks good.
We even checked the IIS logs using specific timings and the HTTP status code for such a transaction is 200. This is not a timeout issue as the database insert takes milliseconds to complete.
Is there anything else that I can check on my side?
Could it be something on the customer's end?
Appreciate any guidance that you can give :-)
There are many possible reasons causing this problem, such as setting up the below code segment before invoking the web service.
System.Net.ServicePointManager.Expect100Continue = false;
Increasing both Timeout and ReadWriteTimeout of request, and assigning the keepAlive=false.
Please refer to the below links for more details.
System.Net.WebException: The underlying connection was closed: The connection was closed unexpectedly
https://social.msdn.microsoft.com/Forums/en-US/246ffc07-1cab-44b5-b529-f1135866ebca/exception-quotthe-underlying-connection-was-closed-the-connection-was-closed-unexpectedlyquot?forum=netfxnetcom
I am new to Amazon Web Services Lambda functions, and I decided to write a small function to ping a website. We have a VPC subnet and security group already set up, which I applied to my function when I published it.
When I run this small function in debug mode through Visual Studio it works okay, but once I publish it to AWS Lambda it fails and I get an exception:
System.Net.NetworkInformation.PingException: An exception occurred during a Ping request. ---> System.PlatformNotSupportedException: The system's ping utility could not be found.\n at System.Net.NetworkInformation.Ping.SendWithPingUtility(IPAddress address, Byte[] buffer, Int32 timeout, PingOptions options)\n at System.Net.NetworkInformation.Ping.SendPingAsyncCore(IPAddress address, Byte[] buffer, Int32 timeout, PingOptions options)\n at System.Net.NetworkInformation.Ping.GetAddressAndSendAsync(String hostNameOrAddress, Int32 timeout, Byte[] buffer, PingOptions options)\n --- End of inner exception stack trace ---\n at System.Net.NetworkInformation.Ping.GetAddressAndSendAsync(String hostNameOrAddress, Int32 timeout, Byte[] buffer, PingOptions options)\n at TestPingWebsite.Function.FunctionHandler(String input, ILambdaContext context) in C:\Users\GWhite\source\TestingWebsite\TestPingWebsite\TestPingWebsite\Function.cs:line 26
Can anybody help me understand why this error occurs please?
AWS lambda does not support outbound ICMP (the protocol that drives ping) traffic. In addition to this, the underlying infrastructure to create the ping messages isnt supported either.
From the AWS docs (https://aws.amazon.com/lambda/faqs/):
Q: What restrictions apply to AWS Lambda function code?
Lambda attempts to impose as few restrictions as possible on normal
language and operating system activities, but there are a few
activities that are disabled: Inbound network connections are blocked
by AWS Lambda, and for outbound connections only TCP/IP and UDP/IP
sockets are supported, and ptrace (debugging) system calls are
blocked. TCP port 25 traffic is also blocked as an anti-spam measure.
I am using a NetworkStream to keep an open TCP/IP connection that messages can be sent across. I receive a message, process it, and then return an ACK. I am working with a site where occasionally I receive the message, but when I go to send the ACK, I get an IOException. Sometimes this lasts for only one or two messages (I can receive the next message), and other times it continues until the service is stopped and restarted.
Below is the code for my NetworkStream without any of the processing:
using (NetworkStream stream = client.GetStream())
{
stream.ReadTimeout = ReadTimeout;
...
if (stream.CanRead && stream.DataAvailable)
bytesRead = stream.Read(receivedBytes, 0, BufferSize);
...
stream.Write(ack, 0, ack.Length);
}
Note that the code loops between reading new messages and sending the ACK.
When I call stream.Write, I will sometimes get the following exception:
System.IO.IOException: Unable to write data to the transport connection: An established connection was aborted by the software in your host machine. ---> System.Net.Sockets.SocketException: An established connection was aborted by the software in your host machine
at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
--- End of inner exception stack trace ---
at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
at Framework.Communication.Model.Server.AcceptMessage(IAsyncResult ar)
I have looked this message up and found several sources for it for several different communication methods. It sounds like this is a very generic exception that does not really tell what is happening.
Does anyone know what would cause this, or where I could start looking to narrow down the solution?
You are right to think this is a generic problem, so you will have to be more defensive.
Somethings to consider
When an application closes a socket the right way, it sends a message containing 0 bytes.
In some cases you may get a SocketException indicating something went wrong.
In a third situation, the remote party is no longer connected (for instance by unplugging the network cable) without any communication between the two parties.
If this happens, you'll have to write data to the socket in order to detect that you can no longer reach the remote party. This is why keep-alive mechanisms were invented - they check every so often whether they can still communicate with the other side.
Note : None of this explains the intermittent nature of part of the
question (if i'm reading it correctly)
So lets see what the documentation says
NetworkStream.Write Method (Byte[], Int32, Int32)
IOException
There was a failure while writing to the network.
An error occurred when accessing the socket. See the Remarks section for more information.
Remarks
If you receive a SocketException, use the
SocketException.ErrorCode property to obtain the specific error
code, and refer to the Windows Sockets version 2 API error code
documentation in MSDN for a detailed description of the error.
So in my mind, as mentioned before you need to be a bit more defensive.
Check for 0 bytes.
Check for errors.
I you get an error make sure you are checking the error codes
In either of these cases, its probably good practice (and makes sense) to restart the connection again, log the failure, and assume the connection has been closed (or abnormally closed)
Additional resources
Detect closed network connection
**** Detection of Half-Open (Dropped) Connections **** Stephen Cleary
I've experienced some random TCP disconnection in production environment: A high load server in Windows Embedded and a client in Windows XP sit in a LAN, connected by long lived TCP via a Switcher, most data flow is from server to client.
The server side core code is straight forward and I simplified as:
try
{
TcpListener tcpListener = new TcpListener(System.Net.IPAddress.Any, 8190);
tcpListener.Start();
var tcpClient = tcpListener.AcceptTcpClient();
var ns = tcpClient.GetStream();
ns.Write(bytes);
}
catch (Exception ex)
{
}
the random exception in that catch is:
System.IO.IOException: Unable to write data to the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
I understand this exception as client has no response (some underlying TCP level ACK or sth) to certain TCP request from server, but the problem is in some cases, the exception just raised up in 4 or 5 seconds from that ms.WriteTo(ns); called.
So the question 1, since I didn't specify any timeout values for TcpClient object, why the Write timeout so short? possible the TCP have the dynamic timeout settings when connection established(based on Round Trip Time or sth), but should be longer(I guess).
==============================
I then write a small program to simulate the situation, I periodically send data from server to client, suddenly I unplug the cable in client machine, what surprise me is the ns.Write(bytes) in server side still can be called smoothly like an async call does, but not the expected blocking call and exception would pop up right after I unplugged the cable.
SO the question 2, if the Write behaves like this, then we never know what data has been successfully send to other side, and this is why we use TCP.
===============================
with cable unplugged, I still keep Write data, I noticed when the data I send reached certain amount (probably 100 bytes), and then the offline time reached 30 seconds, exception raised and also confused me:
System.IO.IOException:
Unable to write data to the transport connection: An existing
connection was forcibly closed by the remote host. --->
System.Net.Sockets.SocketException: An existing connection was
forcibly closed by the remote host at
System.Net.Sockets.Socket.Send(Byte[] buffer, Int32 offset, Int32
size, So cketFlags socketFlags) at
System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset,
Int32 size)
If I just send few bytes, there's no exception at all forever.
SO the question 3, TCP data failed to reach other side might not raise any notification in sender side?
And the question 4, who closed the connection in the exception described? I expected a time out exception here since client side totally disappear.