When I make POST and PUT requests to my ServiceStack services (running standalone with an HTTP listener at the moment) I sometimes find that the request will work, and sometimes my client (HTTPie) will hang. After a while I'll see some info in the Output window in Visual Studio:-
A first chance exception of type 'System.Net.HttpListenerException' occurred in System.dll
A first chance exception of type 'System.Runtime.Serialization.SerializationException' occurred in ServiceStack.dll
A first chance exception of type 'System.Net.HttpListenerException' occurred in System.dll
A first chance exception of type 'System.Net.HttpListenerException' occurred in ServiceStack.dll
A first chance exception of type 'System.Net.HttpListenerException' occurred in ServiceStack.dll
A first chance exception of type 'System.Runtime.Serialization.SerializationException' occurred in ServiceStack.dll
A first chance exception of type 'System.ObjectDisposedException' occurred in System.dll
I managed to enable the built-in logging in ServiceStack and it gave more details:-
ERROR: Error occured while Processing Request: Could not deserialize 'application/json; charset=utf-8'
request using WebServices.Dto.Country'
Error: System.Net.HttpListenerException (0x80004005): The I/O operation has been aborted because of either
a thread exit or an application request
at System.Net.HttpRequestStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at System.IO.StreamReader.ReadBuffer()
at System.IO.StreamReader.ReadToEnd()
at ServiceStack.Text.JsonSerializer.DeserializeFromStream(Type type, Stream stream) in C:\src\ServiceStack.Text\
src\ServiceStack.Text\JsonSerializer.cs:line 164
at ServiceStack.ServiceModel.Serialization.JsonDataContractDeserializer.DeserializeFromStream(Type type, Stream
stream) in C:\src\ServiceStack\src\ServiceStack.Common\ServiceModel\Serialization\JsonDataContractDeserializer.cs:line 86
at ServiceStack.WebHost.Endpoints.Support.EndpointHandlerBase.CreateContentTypeRequest(IHttpRequest httpReq,
Type requestType, String contentType) in C:\src\ServiceStack\src\ServiceStack\WebHost.Endpoints\Support\
EndpointHandlerBase.cs:line 100,
Exception: Could not deserialize 'application/json; charset=utf-8' request using WebServices.Dto.Country'
Error: System.Net.HttpListenerException (0x80004005): The I/O operation has been aborted because of either a thread
exit or an application request
at System.Net.HttpRequestStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at System.IO.StreamReader.ReadBuffer()
at System.IO.StreamReader.ReadToEnd()
at ServiceStack.Text.JsonSerializer.DeserializeFromStream(Type type, Stream stream) in C:\src\ServiceStack.Text\src\
ServiceStack.Text\JsonSerializer.cs:line 164
at ServiceStack.ServiceModel.Serialization.JsonDataContractDeserializer.DeserializeFromStream(Type type, Stream
stream) in C:\src\ServiceStack\src\ServiceStack.Common\ServiceModel\Serialization\JsonDataContractDeserializer.cs:line 86
at ServiceStack.WebHost.Endpoints.Support.EndpointHandlerBase.CreateContentTypeRequest(IHttpRequest httpReq,
Type requestType, String contentType) in C:\src\ServiceStack\src\ServiceStack\WebHost.Endpoints\Support\EndpointHandlerBase.cs:line 100
ERROR: Could not WriteTextToResponse: An operation was attempted on a nonexistent network connection, Exception:
An operation was attempted on a nonexistent network connection
ERROR: Could not WriteTextToResponse: An operation was attempted on a nonexistent network connection, Exception:
An operation was attempted on a nonexistent network connection
INFO: Failed to write error to response: {0}, Exception: An operation was attempted on a nonexistent network connection
I am making my test requests like this:-
http --json post "http://localhost:1337/countries/" DefaultHouse_Id=2 Name=Denmark
This exact attempt works some time and fails others.
I'm using the latest ServiceStack from NuGet with .NET4 and VS 2010.
It appears that the request does not reach my own code at all.
Does anyone know what could cause this, or how I can debug it?
Edit: Not sure if it's related or not, but sometimes I also get this:-
{
"ResponseStatus":{
"ErrorCode":"TypeLoadException",
"Message":"Could not load type 'ServiceStack.ServiceInterface.HttpRequestApplyToExtensions' from assembly 'ServiceStack.ServiceInterface, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null'.",
"StackTrace":" at RulesLayer.AuditFilterAttribute.Execute(IHttpRequest req, IHttpResponse res, Object requestDto)\n at ServiceStack.ServiceInterface.RequestFilterAttribute.RequestFilter(IHttpRequest req, IHttpResponse res, Object requestDto) in C:\\src\\ServiceStack\\src\\ServiceStack.ServiceInterface\\RequestFilterAttribute.cs:line 41\n at ServiceStack.WebHost.Endpoints.EndpointHost.ApplyRequestFilters(IHttpRequest httpReq, IHttpResponse httpRes, Object requestDto) in C:\\src\\ServiceStack\\src\\ServiceStack\\WebHost.Endpoints\\EndpointHost.cs:line 303\n at ServiceStack.WebHost.Endpoints.RestHandler.ProcessRequest(IHttpRequest httpReq, IHttpResponse httpRes, String operationName) in C:\\src\\ServiceStack\\src\\ServiceStack\\WebHost.Endpoints\\RestHandler.cs:line 63"
}
}
It appears AVG Anti-Virus Business was the culprit. I knew I hated anti-virus software for a reason!
I discovered that the code worked fine on another machine, which told me that it was something specific to my dev box setup. I tried stripping the machine back to bare minimum, removing unused software, VS extensions, plugins etc. I tried both IIS Express and self-hosted modes for ServiceStack. I tried moving my code out of Dropbox. Nothing made any difference.
When I finally disabled AVG, the problem just vanished and several hours later it's still working perfectly.
To debug, nothing beats downloading the project source code and debugging the framework source code itself. I recommend trying to replicate this behavior in an integration test so we can find out what's causing it. See ServiceStack's http listener integration tests for examples on how to do this.
To enable logging in ServiceStack you just need set the log factory to be the provider of your choice before you start/initialize the AppHost.
LogManager.LogFactory = new ConsoleLogFactory(typeof(Program));
Related
I have a .Net web service running on a Windows 2012 64-bit application server, IIS 8.
An issue was raised by our customer a number of weeks back and we are finding it very difficult to diagnose where the problem lies!
The customer calls a method on our web service passing an XML data structure. The web method simply inserts that xml data into a SQL Server database table. The return from the web method is a string "OK".
60% of the time the customer reports no issues. But for the other 40%, they are receiving the error...
"The underlying connection was closed: The connection was closed
unexpectedly."
Here's a more detailed trace from the customer's end...
Message: The underlying connection was closed: The connection was
closed unexpectedly. HResult: -2146233079 Source: System StackTrace:
at System.Net.HttpWebRequest.GetResponse() Message: Unable to read
data from the transport connection: The connection was closed.
HResult: -2146232800 Source: System StackTrace: at
System.Net.ConnectStream.InternalWrite(Boolean async, Byte[] buffer,
Int32 offset, Int32 size, AsyncCallback callback, Object state) at
System.Net.ConnectStream.Write(Byte[] buffer, Int32 offset, Int32
size)
Now here's the dilemma. For those failures, we did actually receive the XML data! I have even modified the web service to write to a log file just before the method returns the "OK" value. All looks good.
We even checked the IIS logs using specific timings and the HTTP status code for such a transaction is 200. This is not a timeout issue as the database insert takes milliseconds to complete.
Is there anything else that I can check on my side?
Could it be something on the customer's end?
Appreciate any guidance that you can give :-)
There are many possible reasons causing this problem, such as setting up the below code segment before invoking the web service.
System.Net.ServicePointManager.Expect100Continue = false;
Increasing both Timeout and ReadWriteTimeout of request, and assigning the keepAlive=false.
Please refer to the below links for more details.
System.Net.WebException: The underlying connection was closed: The connection was closed unexpectedly
https://social.msdn.microsoft.com/Forums/en-US/246ffc07-1cab-44b5-b529-f1135866ebca/exception-quotthe-underlying-connection-was-closed-the-connection-was-closed-unexpectedlyquot?forum=netfxnetcom
I am currently upgrading dot net framework in my application from 4.5.2 to 4.7.2 to support TLS1.2.
And I am getting an error The underlying connection was closed: An unexpected error occurred on a receive. on calling asmx web service on remote machine.
Below is the complete stack trace
Error : The underlying connection was closed: An unexpected
error occurred on a receive. Stack Trace : at
System.Web.Services.Protocols.WebClientProtocol.GetWebResponse(WebRequest
request) at
System.Web.Services.Protocols.HttpWebClientProtocol.GetWebResponse(WebRequest
request) at
System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String
methodName, Object[] parameters) at
ProCard.RTMS.ClientProxy.TS2IPAccount.Proxy.HsAccountService.GetIssuedCards(GetIssuedCardsRequest
GetIssuedCardsRequest) in
C:\Upgrade3\healthcare\ClientProxy_Comp\ProCard.RTMS.ClientProxy.Template\TS2IPAccount\ProCard.RTMS.ClientProxy.TS2IPAccount\proxy\TS2IPAccountClientProxy.cs:line
863 at
ProCard.RTMS.ClientProxy.TS2IPAccount.TS2IPAccountProxy.GetIssuedCards(String
clientMsgID, String primeBankNumber, String cardNumber) in
C:\Upgrade3\healthcare\ClientProxy_Comp\ProCard.RTMS.ClientProxy.Template\TS2IPAccount\ProCard.RTMS.ClientProxy.TS2IPAccount\TS2IPAccountProxy.cs:line
2636 Message :
ProCard.RTMS.ClientProxy.TS2IPAccount.TS2IPAccountProxyERROR (probably
from RTMS itself):System.Net.WebException: The underlying connection
was closed: An unexpected error occurred on a receive. --->
System.ComponentModel.Win32Exception: The client and server cannot
communicate, because they do not possess a common algorithm at
System.Net.SSPIWrapper.AcquireCredentialsHandle(SSPIInterface
SecModule, String package, CredentialUse intent, SecureCredential scc)
at
System.Net.Security.SecureChannel.AcquireCredentialsHandle(CredentialUse
credUsage, SecureCredential& secureCredential)
Below is the code where I am getting the error.
public GetIssuedCardsInfo GetIssuedCards
{
HsAccountService proxy = new HsAccountService();
GetIssuedCardsResponse resp = new GetIssuedCardsResponse();
resp = proxy.GetIssuedCards(req);
}
So I am getting error at the response.
When I am sending the soap request using SOAP Ui tool and calling our remote asmx web service I am getting the response correctly but I am not getting the response from our application.
I am currently stuck and not able to find the solution for this particular issue. Please help.
Upon receiving the request on the service side, I always retrieve the content by calling ReadAsStringAsync on the HttpContent object. From the service logs I can see that many times on line HttpContent.ReadAsStringAsync, the service raises following exception
System.Net.Http.HttpRequestException: Error while copying content to a stream. ---> System.IO.IOException ---> System.Net.HttpListenerException: The I/O operation has been aborted because of either a thread exit or an application request
Does anyone know why would I see this exception? and how to resolve this?
thx
I have a windows application (actually an Outlook addin) that calls an asmx web service (yes, it's legacy, but the company is reluctant to move away from it & re-develop everything...) to get some data. The return value is a DataTable. I am calling the web service by adding is as a web service reference.
This has been working correctly for all clients, but recently one client reported an issue where calling the web service throws the following errors (on two separate calls):
Exception Type: InvalidOperationException
Exception Message: There is an error in XML document (23, 93530).
Inner Ex Type: IOException
Inner Ex Message: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.
StackTrace: at System.Xml.Serialization.XmlSerializer.Deserialize(XmlReader xmlReader, String encodingStyle, XmlDeserializationEvents events)
at System.Xml.Serialization.XmlSerializer.Deserialize(XmlReader xmlReader, String encodingStyle)
at System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage message, WebResponse response, Stream responseStream, Boolean asyncCall)
at System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String methodName, Object[] parameters) at
Exception Type: InvalidOperationException
Exception Message: There is an error in XML document (4426, 32).
Inner Ex Type: IOException
Inner Ex Message: The operation has timed out.
StackTrace: at System.Xml.Serialization.XmlSerializer.Deserialize(XmlReader xmlReader, String encodingStyle, XmlDeserializationEvents events)
at System.Xml.Serialization.XmlSerializer.Deserialize(XmlReader xmlReader, String encodingStyle)
at System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage message, WebResponse response, Stream responseStream, Boolean asyncCall)
at System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String methodName, Object[] parameters) at
I am able to rule out that my application is generating the timeout as I am setting the web service class' TimeOut property to 2hrs before the call, and the next request has come through to the server in about 20-30 minutes.
Going through the IIS log, all the requests that have failed for the client have a http status of 200, but also a win32-status of 995.
I have been trying to replicate this error and get the reason why it is occurring without any success for the better part of the last 3 days (I need to find the exact reason - company policy :( :P). I need to know why this occurs and what is the solution or best work-around for this issue.
I have also posted this on the MSDN forum here
Can anyone help me with this? Any help is greatly appreciated. Thanks!
Cheers!
you need to check to see if your hitting a connection limit ask the servers closing it
as the other party to check web.config
<system.net>
<connectionManagement>
<add address="*" maxconnection="40"/>
</connectionManagement>
</system.net>
I have the following WCF client code:
string add = String.Format("http://localhost:{0}/ServiceRequestRest",accessPort);
var cf = new ChannelFactory<IServiceRequestRest>(new WebHttpBinding(), add);
cf.Endpoint.Behaviors.Add(new WebHttpBehavior());
busService = cf.CreateChannel();
busService.DoMyStuff("hello",null);
When I have "first chance exception break" activated, that last line of code throws a series of exceptions. The first exception is this:
System.InvalidOperationException occurred
Message="Envelope Version 'EnvelopeNone (http://schemas.microsoft.com/ws/2005/05/envelope/none)' does not support adding Message Headers."
Source="System.ServiceModel"
StackTrace:
at System.ServiceModel.Channels.MessageHeaders.ValidateHeaderKind(HeaderKind headerKind)
I am not aware that I am trying to "add Message Headers", but I choose to continue execution and immediately get the next exception:
System.Net.WebException occurred
Message="The underlying connection was closed: The connection was closed unexpectedly."
Source="System"
StackTrace:
at System.Net.HttpWebRequest.GetResponse()
With all that said, I'm having difficulty understanding what the problem is. Any clues?
Update: The return type of my REST call could not be serialized, as I learned from another site. That alone could cause this problem, but after I fixed it I still had the same problem. When I use a regular catch block for the exception, all I get is "the connection was closed unexpectedly." What else could I be missing?
Update2: I think I found the final problem. I was getting interference from Skype. The code is working now. :)
The main problem that I see is that you are trying to use a WCF Channel to communicate using WebHttpBinding.
IMHO the entire purpose of WebHttpBinding was so that the client did not have to use the WCF stack to communicate with it. You simply use any HTTP stack to make the requests.