We received the following ConnectionsQuotaExceeded message from Azure Service Bus:
Exception
EventName="MessageEvent" Message="One or more unknown errors occurred communicating with the Service Bus.
Exception Type: QuotaExceededException
Exception Depth: 0
Message: ConnectionsQuotaExceeded for namespace production.
Stack Trace
Stack Trace: at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)
at Microsoft.ServiceBus.Messaging.QueueClient.RetryAcceptMessageSessionAsyncResult.End(IAsyncResult r)
at Microsoft.ServiceBus.Messaging.QueueClient.EndAcceptMessageSession(IAsyncResult result)
at System.Threading.Tasks.TaskFactory 1.FromAsyncCoreLogic(IAsyncResult iar, Func 2 endFunction, Action 1 endAction, Task 1 promise, Boolean requiresSynchronization)
End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Platform.ServiceBus.Utilities.ResilientQueueClient.<>c__DisplayClass23.<b__22>d__25.MoveNext() in d:\vsoagents\18\s\Cloud.ServiceBus.Utilities\ResilientQueueClient.cs:line 0
Exception Type: FaultException`1
Exception Depth: 1
Message: ConnectionsQuotaExceeded for namespace production.
Stack Trace:
Server stack trace:
at Microsoft.ServiceBus.Messaging.Sbmp.DuplexRequestBindingElement.DuplexRequestSessionChannel.ThrowIfFaultMessage(Message wcfMessage)
at Microsoft.ServiceBus.Messaging.Sbmp.DuplexRequestBindingElement.DuplexRequestSessionChannel.HandleMessageReceived(IAsyncResult result)
Problem
The quotas are outlined in this document. The exception appears to refer to this Connection Quota:
Number of concurrent connections on a namespace.
Subsequent requests for additional connections will be rejected and an exception will be received by the calling code.
REST operations do not count towards concurrent TCP connections.
NetMessaging: 1,000
AMQP: 5,000
My problem is we are unable to isolate our exact usage of this quota , likely the number of AMQP connections.
It seems these thresholds are invisible to users until the limitation is reached causing an exception like this. From a capacity planning point of view this make it difficult to understand when scaling will be required on any related resource.
Looking at the Azure portal UI performance for Service Bus the following metrics are available:
Incoming Messages
Internal Service Errors
Length
Server Busy Errors
Size
Successful Requests
Other Errors
Total Requests
Importantly no connections, no AMQP threads, infact none of the quota thresholds detailed in the article ref.
Question
Is there a method via the Azure API or another Azure UI to examine and alert on the quotas documented in the Service Bus Quotas article or at the very least the number of AMQP connections to a service bus namespace ?
Related
I have a function app that sends some data to an Azure event hub, but I get the below exception thrown at certain instances.
System.TimeoutException: Creation of RequestResponseAmqpLink did not complete in 0 milliseconds.
at Microsoft.Azure.Amqp.Singleton1.GetOrCreateAsync(TimeSpan timeout) at Microsoft.Azure.Amqp.TaskHelpers.EndAsyncResult(IAsyncResult asyncResult) at Microsoft.Azure.Amqp.IteratorAsyncResult1.StepCallback(IAsyncResult result)
--- End of stack trace from previous location where exception was thrown ---
Is there a way someone can mitigate this issue? Thanks
For more clarifications i am using EventHubClient.send(EventDataBatch) method
I have a web API(service) whose target framework is .net core 2.0.
This service is calling a custom identity server 4 endpoint using HttpClient by using the 'using' statement. This call always works fine on the development/UAT server.
But on all production servers, it fails intermittently and throws the following exception:
One or more errors occurred. (An error occurred while sending the
request.) Exception: System.AggregateException: One or more errors
occurred. (An error occurred while sending the request.) --->
System.Net.Http.HttpRequestException: An error occurred while sending
the request. ---> System.Net.Http.WinHttpException: The operation
timed out at
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at
System.Threading.Tasks.RendezvousAwaitable`1.GetResult() at
System.Net.Http.WinHttpHandler.d__105.MoveNext() ---
End of inner exception stack trace --- at
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task) at
System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
at
System.Net.Http.HttpClient.d__58.MoveNext()
--- End of inner exception stack trace --- at System.Threading.Tasks.Task`1.GetResultCore(Boolean
waitCompletionNotification) at Get(String a, String b)
I went through related stack-overflow posts, some suggested making HttpClient static as it might throw socket exhaustion error. But I am not getting that error either I am able to reproduce this issue on a development server. I created a console application and called this service 64k times, but still, it worked properly.
How to stop outbound HTTP connections from timing out
Asp.net Core HttpClient has many TIME_WAIT or CLOSE_WAIT connections
Can someone please tell me, what can be the reason for this intermittent error only on the production server or is there any way I can reproduce this issue on the dev server. I have tried by making many calls to service, but still unable to reproduce the error.
Our WCF client normally invokes a web service. All deployments of the client invoke the same web service. In most deployments it makes the call without any issues, except in one instance it reports the following exception:
System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel. ---> System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure.
at System.Net.TlsStream.EndWrite(IAsyncResult asyncResult)
at System.Net.PooledStream.EndWrite(IAsyncResult asyncResult)
at System.Net.ConnectStream.WriteHeadersCallback(IAsyncResult ar)
--- End of inner exception stack trace ---
at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)
--- End of inner exception stack trace ---
What could be different with this one installation?
We tested the URL that the client uses to make the call to the web service in the browser. The web service returns a response ok in Firefox but not in IE.
Have you seen this article yet? It contains fairly good tips for troubleshooting these issues: http://blogs.msdn.com/b/jpsanders/archive/2009/09/16/troubleshooting-asp-net-the-remote-certificate-is-invalid-according-to-the-validation-procedure.aspx?Redirected=true
For example, you'll need to check that the particular machine has all the root and intermediate certificates properly deployed.
I found the solution - there was a new web service server certificate installed with SHA256 and Windows 2003 Server requires a hotfix to support it:
http://support.microsoft.com/kb/938397
I have a legacy WCF service using FluentNHibernate (along side EF and stored procs). The service was working fine for over an year. Recently we change the apppool recycle settings from default to make apppool recycle at 10 AM every day. Since that change, at least twice, my WCF service has failed with following exception
FluentNHibernate.Cfg.FluentConfigurationException: An invalid or incomplete configuration was used while creating a SessionFactory. Check PotentialReasons collection, and InnerException for more detail.
---> FluentNHibernate.Cfg.FluentConfigurationException: An invalid or incomplete configuration was used while creating a SessionFactory. Check PotentialReasons collection, and InnerException for more detail.
---> NHibernate.MappingException: Could not configure datastore from input stream (XmlDocument) ---> System.NullReferenceException: Object reference not set to an instance of an object.
at NHibernate.Cfg.MappingsQueue.GetNextAvailableResource()
at NHibernate.Cfg.Configuration.AddInputStream(Stream xmlInputStream, String name)
--- End of inner exception stack trace ---
at NHibernate.Cfg.Configuration.LogAndThrow(Exception exception)
at NHibernate.Cfg.Configuration.AddInputStream(Stream xmlInputStream, String name)
at NHibernate.Cfg.Configuration.AddDocument(XmlDocument doc, String name)
at FluentNHibernate.PersistenceModel.Configure(Configuration cfg)
at FluentNHibernate.Cfg.MappingConfiguration.Apply(Configuration cfg)
at FluentNHibernate.Cfg.FluentConfiguration.BuildConfiguration()
--- End of inner exception stack trace ---
at FluentNHibernate.Cfg.FluentConfiguration.BuildConfiguration()
at FluentNHibernate.Cfg.FluentConfiguration.BuildSessionFactory()
--- End of inner exception stack trace ---
at FluentNHibernate.Cfg.FluentConfiguration.BuildSessionFactory()
A few interesting things to note
Both times, the error has started appearing few seconds after the apppool recycle which makes me believe this is closely related to that
This does not happen after every apppool recycle
This does not happen in our test environments where apppool recycle setting is IIS default
This is very difficult to reproduce locally. I have looked at following SO questions regarding the same exceptions but they are in a completely different context and the authors of those questions are getting that exception consistently which is not the case here
Runtime error when trying to run Fluent NHibernate tutorial example
Fluent nHibernate error - Could not compile the mapping document: (XmlDocument)
After hours of investigation, it came down to be an issue of multiple threads trying to create session factory. I would have thought that NH session factory is thread-safe but sadly it isn't and I had to put int code to make it thread-safe.
Sorry if the Title is a little vague but the truth is I do not understand my problem, I have never come across this before and I havent done anything out of the Ordinary.
Right I have a basic Silverlight 4 Application using the Business Application Template. I have also created a WCF Service Application Project which is used to simply return an Observable collection for some entries in my Database,
I published this to my Local computer which I then referenced in my SL4 app and I get this exception
[System.ServiceModel.CommunicationException: The remote server returned an error: NotFound. ---> System.Net.WebException: The remote server returned an error: NotFound. ---> System.Net.WebException: The remote server returned an error: NotFound.
at System.Net.Browser.BrowserHttpWebRequest.InternalEndGetResponse(IAsyncResult asyncResult)
at System.Net.Browser.BrowserHttpWebRequest.<>c__DisplayClass5. <EndGetResponse>b__4(Object sendState)
at System.Net.Browser.AsyncHelper.<>c__DisplayClass4.<BeginOnUI>b__0(Object sendState)
--- End of inner exception stack trace ---
at System.Net.Browser.AsyncHelper.BeginOnUI(SendOrPostCallback beginMethod, Object state)
at System.Net.Browser.BrowserHttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
at System.ServiceModel.Channels.HttpChannelFactory.HttpRequestChannel.HttpChannelAsyncRequest. CompleteGetResponse(IAsyncResult result)
--- End of inner exception stack trace ---
at System.ServiceModel.AsyncResult.End[TAsyncResult](IAsyncResult result)
at System.ServiceModel.Channels.ServiceChannel.EndCall(String action, Object[] outs, IAsyncResult result)
at System.ServiceModel.ClientBase`1.ChannelBase`1.EndInvoke(String methodName, Object[] args, IAsyncResult result)
at RMServiceRIA.rService.Service1Client.Service1ClientChannel.EndgetBuildings(IAsyncResult result)
at RMServiceRIA.rService.Service1Client.RMServiceRIA.rService.IService1.EndgetBuildings(IAsync Result result)
at RMServiceRIA.rService.Service1Client.OnEndgetBuildings(IAsyncResult result)
at System.ServiceModel.ClientBase`1.OnAsyncCallCompleted(IAsyncResult result)}
I have got a clientaccess policy and crossdomain policy which for the purpose of testing is set to allow anything access,
Client Access Policy
Cross Domain
any ideas on how to fix this problem, I debugged my WCF Service and it receives the call, the method executed reaches its return statement, yet my SL4 never even receives the result, as soon as my WCF service has the result I get an exception thrown within my SL4 app, to me it just seems that the connection gets closed but I dont know and I have tried to research but where do you start with something you dont fully understand? I found many people with the same sort of error message but WCF is so damn cryptic... I did try and enable the debugging information inside of my webconfig. I have also tried to host the WCF service on my Web server but no luck, I have several WCF Services running on that with a similar set up so I am unsure why this isnt working
Web Config for WCF Service
client config
Thanks for having a look, and if you need any more info just ask I'll be happy to provide you with it,
Your best bet is to turn on WCF Tracing so you can see what is going on.
http://software.intel.com/en-us/blogs/2010/02/03/adventures-with-silverlight-and-wcf-the-remote-server-returned-an-error-notfound/
Most likely there is an issue with serializing something. Not found is just the Silverlight version of a 500 error.