I have a web API(service) whose target framework is .net core 2.0.
This service is calling a custom identity server 4 endpoint using HttpClient by using the 'using' statement. This call always works fine on the development/UAT server.
But on all production servers, it fails intermittently and throws the following exception:
One or more errors occurred. (An error occurred while sending the
request.) Exception: System.AggregateException: One or more errors
occurred. (An error occurred while sending the request.) --->
System.Net.Http.HttpRequestException: An error occurred while sending
the request. ---> System.Net.Http.WinHttpException: The operation
timed out at
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at
System.Threading.Tasks.RendezvousAwaitable`1.GetResult() at
System.Net.Http.WinHttpHandler.d__105.MoveNext() ---
End of inner exception stack trace --- at
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task) at
System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
at
System.Net.Http.HttpClient.d__58.MoveNext()
--- End of inner exception stack trace --- at System.Threading.Tasks.Task`1.GetResultCore(Boolean
waitCompletionNotification) at Get(String a, String b)
I went through related stack-overflow posts, some suggested making HttpClient static as it might throw socket exhaustion error. But I am not getting that error either I am able to reproduce this issue on a development server. I created a console application and called this service 64k times, but still, it worked properly.
How to stop outbound HTTP connections from timing out
Asp.net Core HttpClient has many TIME_WAIT or CLOSE_WAIT connections
Can someone please tell me, what can be the reason for this intermittent error only on the production server or is there any way I can reproduce this issue on the dev server. I have tried by making many calls to service, but still unable to reproduce the error.
Related
All Azure functions work fine in "MySoftware 2.0.14". I had to redeploy all azure functions and change the addresses in the software and release the new version "MySoftware 2.0.18". Everything worked good but for about 3-10% of my customers stopped working all functions.
I had a TeamViewer session with a lot of these guys and the functions behave like they just don't exist. Like when I change the function address to some random string. It is in both cases throws "An error occurred while sending the request."
The full exception of these guys:
System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a send. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
at System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult)
at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)
--- End of inner exception stack trace ---
at System.Net.TlsStream.EndWrite(IAsyncResult asyncResult)
at System.Net.ConnectStream.WriteHeadersCallback(IAsyncResult ar)
--- End of inner exception stack trace ---
at System.Net.HttpWebRequest.EndGetRequestStream(IAsyncResult asyncResult, TransportContext& context)
at System.Net.Http.HttpClientHandler.GetRequestStreamCallback(IAsyncResult ar)
--- End of inner exception stack trace ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at (TaskAwaiter`1& )
at InstaGTO.Model.ServerCommunication.ServerRequest.♶┩♼﵆⚝┚⚶╇╽♅⚢▚♋⛴╬♽⛣⚒⚧⛍▄☽▄┢┒▷◔┪╨⚿♻䷪♩☈☯╃䷃╓⚨⚾♠◪▒☦☔┻╩﷾⛿﵏┅╰䷚▊┸◞䷠⛂◑♑┾╲◟⚇♞╇┈◢╳│⛖☃☲┼◩▼◢﵅◨█┕╄⚞┨◤◊⛣┦.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at (TaskAwaiter`1& )
at InstaGTO.ViewModel.RegistrationViewModel.◮☶╞☿┿䷯⚥╻╋﮻♭┣▣╖⚇☢﯀☼☝♐⛔⚇☗⛤☣﵂䷤䷦◥╱▩⛃╷⚒▉⚫▇⚓⚺◻▒♓╥䷹║♀●═╺䷚䷵䷲䷾┞☍♶⛎◜▚♥◕♒┉☬▌▀⚻䷜☸◿▫◶⚀├▹⛼╢┤⛤╄⚚♔⛲◱♬╲★◷┸.MoveNext()
I didn't make any changes in the functions neither in the code. I just only deploy them on the new Azure plan and change addresses in the software. I really don't understand why it doesn't work for someone. The insights didn't notice any error but my software just doesn't work for those guys because they cannot sign up, cannot request a new password, register and all stuff on the cloud.
As suggested by Hari, Following https://github.com/Azure/azure-functions-host/issues/3665 and SSL changed to 1.0 did solve the issue.
I am logging the connection status events with the wcf relay, and I'm seeing something like this in the logs.
1/26 06:47:12 ERROR Service Bus ConnectionStatus: 'Reconnecting' Event. [(null)][42]
LastError: System.ServiceModel.CommunicationException: Exception of type 'System.ServiceModel.CommunicationException' was thrown. ---> Microsoft.ServiceBus.Messaging.Amqp.AmqpException: An AMQP error occurred (condition='amqp:unauthorized-access').
--- End of inner exception stack trace ---
This exception doesn't show up in the list on this microsoft page, and the only other post I can find anywhere related to this error message is here. However, that post does not have any recent comments or a resolution or workaround for the issue. Also, the exception doesn't have any stacktrace, so how am I supposed to troubleshoot this error?
I guess as a follow-up, I would ask whether this is anything to actually worry about if the wcf connection is never faulting.
Apparently, the token that the relay keeps refreshing to stay active requires the time on the server to match the azure service that it is connecting with, and if not, this type of error will show up. We were able to fix it by correcting the server time.
We are connecting to a specific website using a TLS connection. We are enforcing TLS 1.2 as the only allowed protocol in the client code. In about 99% of all cases connection setup is OK and shows no errors. However from time to time we receive the following exception:
System.Security.Authentication.AuthenticationException: A call to SSPI failed, see inner exception. ---> System.ComponentModel.Win32Exception: The buffers supplied to a function was too small
--- End of inner exception stack trace ---
at System.Net.Security.SslState.EndProcessAuthentication(IAsyncResult result)
at Fw.Connections.Ssl.Private.ClientInstance.AuthenticateClientCallback(IAsyncResult result);
We have tested the remote server with SSL labs and we receive an A+ with no failures or problems reported, it's not a server under our control so the remote environment is not very well known.. We have tested the client code on Windows server 2012 (datacenter) and Windows7 clients. Both show the error from time to time. Is this a known bug in the .NET framework?
We received the following ConnectionsQuotaExceeded message from Azure Service Bus:
Exception
EventName="MessageEvent" Message="One or more unknown errors occurred communicating with the Service Bus.
Exception Type: QuotaExceededException
Exception Depth: 0
Message: ConnectionsQuotaExceeded for namespace production.
Stack Trace
Stack Trace: at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)
at Microsoft.ServiceBus.Messaging.QueueClient.RetryAcceptMessageSessionAsyncResult.End(IAsyncResult r)
at Microsoft.ServiceBus.Messaging.QueueClient.EndAcceptMessageSession(IAsyncResult result)
at System.Threading.Tasks.TaskFactory 1.FromAsyncCoreLogic(IAsyncResult iar, Func 2 endFunction, Action 1 endAction, Task 1 promise, Boolean requiresSynchronization)
End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Platform.ServiceBus.Utilities.ResilientQueueClient.<>c__DisplayClass23.<b__22>d__25.MoveNext() in d:\vsoagents\18\s\Cloud.ServiceBus.Utilities\ResilientQueueClient.cs:line 0
Exception Type: FaultException`1
Exception Depth: 1
Message: ConnectionsQuotaExceeded for namespace production.
Stack Trace:
Server stack trace:
at Microsoft.ServiceBus.Messaging.Sbmp.DuplexRequestBindingElement.DuplexRequestSessionChannel.ThrowIfFaultMessage(Message wcfMessage)
at Microsoft.ServiceBus.Messaging.Sbmp.DuplexRequestBindingElement.DuplexRequestSessionChannel.HandleMessageReceived(IAsyncResult result)
Problem
The quotas are outlined in this document. The exception appears to refer to this Connection Quota:
Number of concurrent connections on a namespace.
Subsequent requests for additional connections will be rejected and an exception will be received by the calling code.
REST operations do not count towards concurrent TCP connections.
NetMessaging: 1,000
AMQP: 5,000
My problem is we are unable to isolate our exact usage of this quota , likely the number of AMQP connections.
It seems these thresholds are invisible to users until the limitation is reached causing an exception like this. From a capacity planning point of view this make it difficult to understand when scaling will be required on any related resource.
Looking at the Azure portal UI performance for Service Bus the following metrics are available:
Incoming Messages
Internal Service Errors
Length
Server Busy Errors
Size
Successful Requests
Other Errors
Total Requests
Importantly no connections, no AMQP threads, infact none of the quota thresholds detailed in the article ref.
Question
Is there a method via the Azure API or another Azure UI to examine and alert on the quotas documented in the Service Bus Quotas article or at the very least the number of AMQP connections to a service bus namespace ?
Our WCF client normally invokes a web service. All deployments of the client invoke the same web service. In most deployments it makes the call without any issues, except in one instance it reports the following exception:
System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel. ---> System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure.
at System.Net.TlsStream.EndWrite(IAsyncResult asyncResult)
at System.Net.PooledStream.EndWrite(IAsyncResult asyncResult)
at System.Net.ConnectStream.WriteHeadersCallback(IAsyncResult ar)
--- End of inner exception stack trace ---
at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)
--- End of inner exception stack trace ---
What could be different with this one installation?
We tested the URL that the client uses to make the call to the web service in the browser. The web service returns a response ok in Firefox but not in IE.
Have you seen this article yet? It contains fairly good tips for troubleshooting these issues: http://blogs.msdn.com/b/jpsanders/archive/2009/09/16/troubleshooting-asp-net-the-remote-certificate-is-invalid-according-to-the-validation-procedure.aspx?Redirected=true
For example, you'll need to check that the particular machine has all the root and intermediate certificates properly deployed.
I found the solution - there was a new web service server certificate installed with SHA256 and Windows 2003 Server requires a hotfix to support it:
http://support.microsoft.com/kb/938397