We Keep Coding

Why does Quartz .NET 3.0.7 scheduler hangs on DB unavailable error

I'm having an issue with a very simple non-clustered quartz server within a C# console application (run as windows service). It includes a single scheduler, that reads from and writes to a single database, meaning the same group of "qrtz_*" tables.
Occasionally, mostly at the weekend at night, when the company I work at performs DB backups and general maintenance, the scheduler listener I set up throws
QuartzSchedulerListener
Scheduler Error -> MESSAGE: An error occurred while scanning for the next trigger to fire.
Couldn't acquire next trigger: Fatal error encountered during command execution.
at Quartz.Impl.AdoJobStore.JobStoreSupport.d__229.MoveNext() in C:\projects\quartznet\src\Quartz\Impl\AdoJobStore\JobStoreSupport.cs:line 2688 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Quartz.Impl.AdoJobStore.JobStoreSupport.d__262`1.MoveNext() in C:\projects\quartznet\src\Quartz\Impl\AdoJobStore\JobStoreSupport.cs:line 3816 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at Quartz.Impl.AdoJobStore.JobStoreSupport.d__262`1.MoveNext() in C:\projects\quartznet\src\Quartz\Impl\AdoJobStore\JobStoreSupport.cs:line 3831 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Quartz.Impl.AdoJobStore.JobStoreSupport.d__228.MoveNext() in C:\projects\quartznet\src\Quartz\Impl\AdoJobStore\JobStoreSupport.cs:line 2542 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task) at Quartz.Core.QuartzSchedulerThread.d__28.MoveNext() in C:\projects\quartznet\src\Quartz\Core\QuartzSchedulerThread.cs:line 262
I'm assuming due to DB temporarily unavailable.
Then, sometimes, it just hangs and stops executing any trigger - it doesn't update triggers' NEXT_FIRE_TIME and there's nothing I can do but restarting the console app (reconfiguring the scheduler and restarting it).
I've tried increasing dbCOmmandTimeout in the scheduler config connection string but it just looks like it gets stuck on some kind of deadlock.
PS: I have a similar setup, a group of quartz 2.6.1 schedulers reading/writing from several DBs. They get the same errors on DB maintenance but they have never stopped in more then a year of activity.
I've run out of ideas, please let me know whether this occurred to anyone.
Thank you

We had the same problem with Quartz version < 3.1.. There's a changelog: https://github.com/quartznet/quartznet/blob/main/changelog.md
I think you have to update the Quartz to at least 3.1 where some deadlock problem on DB unavailability has been fixed:
There is also a very important bug fix present for lock handling on retries. There was a possibility for a deadlock in database lock handling in some situations.
Hope it will help as it helped me.

Intermittent "The operation timed out" error on production server

I have a web API(service) whose target framework is .net core 2.0.
This service is calling a custom identity server 4 endpoint using HttpClient by using the 'using' statement. This call always works fine on the development/UAT server.
But on all production servers, it fails intermittently and throws the following exception:
One or more errors occurred. (An error occurred while sending the
request.) Exception: System.AggregateException: One or more errors
occurred. (An error occurred while sending the request.) --->
System.Net.Http.HttpRequestException: An error occurred while sending
the request. ---> System.Net.Http.WinHttpException: The operation
timed out at
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at
System.Threading.Tasks.RendezvousAwaitable`1.GetResult() at
System.Net.Http.WinHttpHandler.d__105.MoveNext() ---
End of inner exception stack trace --- at
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task) at
System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
at
System.Net.Http.HttpClient.d__58.MoveNext()
--- End of inner exception stack trace --- at System.Threading.Tasks.Task`1.GetResultCore(Boolean
waitCompletionNotification) at Get(String a, String b)
I went through related stack-overflow posts, some suggested making HttpClient static as it might throw socket exhaustion error. But I am not getting that error either I am able to reproduce this issue on a development server. I created a console application and called this service 64k times, but still, it worked properly.
How to stop outbound HTTP connections from timing out
Asp.net Core HttpClient has many TIME_WAIT or CLOSE_WAIT connections
Can someone please tell me, what can be the reason for this intermittent error only on the production server or is there any way I can reproduce this issue on the dev server. I have tried by making many calls to service, but still unable to reproduce the error.

Azure functions throws "An error occurred while sending the request" for some of my customers

All Azure functions work fine in "MySoftware 2.0.14". I had to redeploy all azure functions and change the addresses in the software and release the new version "MySoftware 2.0.18". Everything worked good but for about 3-10% of my customers stopped working all functions.
I had a TeamViewer session with a lot of these guys and the functions behave like they just don't exist. Like when I change the function address to some random string. It is in both cases throws "An error occurred while sending the request."
The full exception of these guys:
System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a send. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
at System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult)
at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)
--- End of inner exception stack trace ---
at System.Net.TlsStream.EndWrite(IAsyncResult asyncResult)
at System.Net.ConnectStream.WriteHeadersCallback(IAsyncResult ar)
--- End of inner exception stack trace ---
at System.Net.HttpWebRequest.EndGetRequestStream(IAsyncResult asyncResult, TransportContext& context)
at System.Net.Http.HttpClientHandler.GetRequestStreamCallback(IAsyncResult ar)
--- End of inner exception stack trace ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at (TaskAwaiter`1& )
at InstaGTO.Model.ServerCommunication.ServerRequest.♶┩♼﵆⚝┚⚶╇╽♅⚢▚♋﷜⛴﷖╬♽⛣⚒⚧﷈⛍▄￱☽▄┢﷪┒▷◔┪╨⚿﷟♻䷪♩☈☯╃䷃╓⚨⚾♠◪▒☦☔┻╩﷾⛿﵏┅╰䷚▊┸◞䷠￸⛂◑♑┾╲◟⚇♞╇┈﯃◢﷚╳│⛖☃☲┼◩▼◢﵅◨█┕﷮╄⚞┨◤◊⛣┦.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at (TaskAwaiter`1& )
at InstaGTO.ViewModel.RegistrationViewModel.◮☶╞☿﷧┿䷯⚥╻╋﮻♭┣▣╖⚇☢﯀☼￳☝♐﯃⛔⚇﷎☗⛤☣﵂䷤䷦◥╱▩⛃╷⚒﷑▉⚫▇⚓⚺◻▒♓╥䷹║♀●═╺䷚䷵䷲﯈䷾﷬┞☍♶⛎﷡◜▚♥◕♒┉☬▌▀⚻䷜☸◿▫﷚◶⚀├▹⛼╢┤⛤╄⚚♔⛲◱♬╲★￴◷┸.MoveNext()
I didn't make any changes in the functions neither in the code. I just only deploy them on the new Azure plan and change addresses in the software. I really don't understand why it doesn't work for someone. The insights didn't notice any error but my software just doesn't work for those guys because they cannot sign up, cannot request a new password, register and all stuff on the cloud.

As suggested by Hari, Following https://github.com/Azure/azure-functions-host/issues/3665 and SSL changed to 1.0 did solve the issue.

Azure Service Bus - Message: ConnectionsQuotaExceeded

We received the following ConnectionsQuotaExceeded message from Azure Service Bus:
Exception
EventName="MessageEvent" Message="One or more unknown errors occurred communicating with the Service Bus.
Exception Type: QuotaExceededException
Exception Depth: 0
Message: ConnectionsQuotaExceeded for namespace production.
Stack Trace
Stack Trace: at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)
at Microsoft.ServiceBus.Messaging.QueueClient.RetryAcceptMessageSessionAsyncResult.End(IAsyncResult r)
at Microsoft.ServiceBus.Messaging.QueueClient.EndAcceptMessageSession(IAsyncResult result)
at System.Threading.Tasks.TaskFactory 1.FromAsyncCoreLogic(IAsyncResult iar, Func 2 endFunction, Action 1 endAction, Task 1 promise, Boolean requiresSynchronization)
End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Platform.ServiceBus.Utilities.ResilientQueueClient.<>c__DisplayClass23.<b__22>d__25.MoveNext() in d:\vsoagents\18\s\Cloud.ServiceBus.Utilities\ResilientQueueClient.cs:line 0
Exception Type: FaultException`1
Exception Depth: 1
Message: ConnectionsQuotaExceeded for namespace production.
Stack Trace:
Server stack trace:
at Microsoft.ServiceBus.Messaging.Sbmp.DuplexRequestBindingElement.DuplexRequestSessionChannel.ThrowIfFaultMessage(Message wcfMessage)
at Microsoft.ServiceBus.Messaging.Sbmp.DuplexRequestBindingElement.DuplexRequestSessionChannel.HandleMessageReceived(IAsyncResult result)
Problem
The quotas are outlined in this document. The exception appears to refer to this Connection Quota:
Number of concurrent connections on a namespace.
Subsequent requests for additional connections will be rejected and an exception will be received by the calling code.
REST operations do not count towards concurrent TCP connections.
NetMessaging: 1,000
AMQP: 5,000
My problem is we are unable to isolate our exact usage of this quota , likely the number of AMQP connections.
It seems these thresholds are invisible to users until the limitation is reached causing an exception like this. From a capacity planning point of view this make it difficult to understand when scaling will be required on any related resource.
Looking at the Azure portal UI performance for Service Bus the following metrics are available:
Incoming Messages
Internal Service Errors
Length
Server Busy Errors
Size
Successful Requests
Other Errors
Total Requests
Importantly no connections, no AMQP threads, infact none of the quota thresholds detailed in the article ref.
Question
Is there a method via the Azure API or another Azure UI to examine and alert on the quotas documented in the Service Bus Quotas article or at the very least the number of AMQP connections to a service bus namespace ?

Communication Exception with my WCF Service SL4 app

Sorry if the Title is a little vague but the truth is I do not understand my problem, I have never come across this before and I havent done anything out of the Ordinary.
Right I have a basic Silverlight 4 Application using the Business Application Template. I have also created a WCF Service Application Project which is used to simply return an Observable collection for some entries in my Database,
I published this to my Local computer which I then referenced in my SL4 app and I get this exception
[System.ServiceModel.CommunicationException: The remote server returned an error: NotFound. ---> System.Net.WebException: The remote server returned an error: NotFound. ---> System.Net.WebException: The remote server returned an error: NotFound.
at System.Net.Browser.BrowserHttpWebRequest.InternalEndGetResponse(IAsyncResult asyncResult)
at System.Net.Browser.BrowserHttpWebRequest.<>c__DisplayClass5. <EndGetResponse>b__4(Object sendState)
at System.Net.Browser.AsyncHelper.<>c__DisplayClass4.<BeginOnUI>b__0(Object sendState)
--- End of inner exception stack trace ---
at System.Net.Browser.AsyncHelper.BeginOnUI(SendOrPostCallback beginMethod, Object state)
at System.Net.Browser.BrowserHttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
at System.ServiceModel.Channels.HttpChannelFactory.HttpRequestChannel.HttpChannelAsyncRequest. CompleteGetResponse(IAsyncResult result)
--- End of inner exception stack trace ---
at System.ServiceModel.AsyncResult.End[TAsyncResult](IAsyncResult result)
at System.ServiceModel.Channels.ServiceChannel.EndCall(String action, Object[] outs, IAsyncResult result)
at System.ServiceModel.ClientBase`1.ChannelBase`1.EndInvoke(String methodName, Object[] args, IAsyncResult result)
at RMServiceRIA.rService.Service1Client.Service1ClientChannel.EndgetBuildings(IAsyncResult result)
at RMServiceRIA.rService.Service1Client.RMServiceRIA.rService.IService1.EndgetBuildings(IAsync Result result)
at RMServiceRIA.rService.Service1Client.OnEndgetBuildings(IAsyncResult result)
at System.ServiceModel.ClientBase`1.OnAsyncCallCompleted(IAsyncResult result)}
I have got a clientaccess policy and crossdomain policy which for the purpose of testing is set to allow anything access,
Client Access Policy
Cross Domain
any ideas on how to fix this problem, I debugged my WCF Service and it receives the call, the method executed reaches its return statement, yet my SL4 never even receives the result, as soon as my WCF service has the result I get an exception thrown within my SL4 app, to me it just seems that the connection gets closed but I dont know and I have tried to research but where do you start with something you dont fully understand? I found many people with the same sort of error message but WCF is so damn cryptic... I did try and enable the debugging information inside of my webconfig. I have also tried to host the WCF service on my Web server but no luck, I have several WCF Services running on that with a similar set up so I am unsure why this isnt working
Web Config for WCF Service
client config
Thanks for having a look, and if you need any more info just ask I'll be happy to provide you with it,

Your best bet is to turn on WCF Tracing so you can see what is going on.
http://software.intel.com/en-us/blogs/2010/02/03/adventures-with-silverlight-and-wcf-the-remote-server-returned-an-error-notfound/
Most likely there is an issue with serializing something. Not found is just the Silverlight version of a 500 error.