I have a legacy WCF service using FluentNHibernate (along side EF and stored procs). The service was working fine for over an year. Recently we change the apppool recycle settings from default to make apppool recycle at 10 AM every day. Since that change, at least twice, my WCF service has failed with following exception
FluentNHibernate.Cfg.FluentConfigurationException: An invalid or incomplete configuration was used while creating a SessionFactory. Check PotentialReasons collection, and InnerException for more detail.
---> FluentNHibernate.Cfg.FluentConfigurationException: An invalid or incomplete configuration was used while creating a SessionFactory. Check PotentialReasons collection, and InnerException for more detail.
---> NHibernate.MappingException: Could not configure datastore from input stream (XmlDocument) ---> System.NullReferenceException: Object reference not set to an instance of an object.
at NHibernate.Cfg.MappingsQueue.GetNextAvailableResource()
at NHibernate.Cfg.Configuration.AddInputStream(Stream xmlInputStream, String name)
--- End of inner exception stack trace ---
at NHibernate.Cfg.Configuration.LogAndThrow(Exception exception)
at NHibernate.Cfg.Configuration.AddInputStream(Stream xmlInputStream, String name)
at NHibernate.Cfg.Configuration.AddDocument(XmlDocument doc, String name)
at FluentNHibernate.PersistenceModel.Configure(Configuration cfg)
at FluentNHibernate.Cfg.MappingConfiguration.Apply(Configuration cfg)
at FluentNHibernate.Cfg.FluentConfiguration.BuildConfiguration()
--- End of inner exception stack trace ---
at FluentNHibernate.Cfg.FluentConfiguration.BuildConfiguration()
at FluentNHibernate.Cfg.FluentConfiguration.BuildSessionFactory()
--- End of inner exception stack trace ---
at FluentNHibernate.Cfg.FluentConfiguration.BuildSessionFactory()
A few interesting things to note
Both times, the error has started appearing few seconds after the apppool recycle which makes me believe this is closely related to that
This does not happen after every apppool recycle
This does not happen in our test environments where apppool recycle setting is IIS default
This is very difficult to reproduce locally. I have looked at following SO questions regarding the same exceptions but they are in a completely different context and the authors of those questions are getting that exception consistently which is not the case here
Runtime error when trying to run Fluent NHibernate tutorial example
Fluent nHibernate error - Could not compile the mapping document: (XmlDocument)
After hours of investigation, it came down to be an issue of multiple threads trying to create session factory. I would have thought that NH session factory is thread-safe but sadly it isn't and I had to put int code to make it thread-safe.
Related
I have a web API(service) whose target framework is .net core 2.0.
This service is calling a custom identity server 4 endpoint using HttpClient by using the 'using' statement. This call always works fine on the development/UAT server.
But on all production servers, it fails intermittently and throws the following exception:
One or more errors occurred. (An error occurred while sending the
request.) Exception: System.AggregateException: One or more errors
occurred. (An error occurred while sending the request.) --->
System.Net.Http.HttpRequestException: An error occurred while sending
the request. ---> System.Net.Http.WinHttpException: The operation
timed out at
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at
System.Threading.Tasks.RendezvousAwaitable`1.GetResult() at
System.Net.Http.WinHttpHandler.d__105.MoveNext() ---
End of inner exception stack trace --- at
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task) at
System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
at
System.Net.Http.HttpClient.d__58.MoveNext()
--- End of inner exception stack trace --- at System.Threading.Tasks.Task`1.GetResultCore(Boolean
waitCompletionNotification) at Get(String a, String b)
I went through related stack-overflow posts, some suggested making HttpClient static as it might throw socket exhaustion error. But I am not getting that error either I am able to reproduce this issue on a development server. I created a console application and called this service 64k times, but still, it worked properly.
How to stop outbound HTTP connections from timing out
Asp.net Core HttpClient has many TIME_WAIT or CLOSE_WAIT connections
Can someone please tell me, what can be the reason for this intermittent error only on the production server or is there any way I can reproduce this issue on the dev server. I have tried by making many calls to service, but still unable to reproduce the error.
Let me try to describe the situation..
Site A:
Is meant is an catch all site, with only 1 library. This is used by an third party software, that send all the items to 1 location with specified information. Behind this is an timerjob, that sends all the items to their respective sites/libraries. (so doesn't stay on Site A)
Site X:
When an new item is added through the timerjob on Site A. An workflow is started, however the workflow is randomly sunspended with a the same error. A manual start on the same item will allways succeed. Thus only the autostart that gets suspended.
Workflow is made in visual studio and is widely used in the application without problems.
The error message:
Details: An unhandled exception occurred during the execution of the
workflow instance. Exception details: System.ApplicationException:
HTTP 401 {"error_description":"The server was unable to process the
request due to an internal error. For more information about the
error, either turn on IncludeExceptionDetailInFaults (either from
ServiceBehaviorAttribute or from the configuration
behavior) on the server in order to send the exception information
back to the client, or turn on tracing as per the Microsoft .NET
Framework SDK documentation and inspect the server trace logs."}
{"x-ms-diagnostics":["3001000;reason=\"There has been an error
authenticating the
request.\";category=\"invalid_client\""],"SPRequestGuid":["c0fb74e8-6b48-0deb-85c4-7fdab324d2f7"],"request-id":["c0fb74e8-6b48-0deb-85c4-7fdab324d2f7"],"X-FRAME-OPTIONS":["SAMEORIGIN"],"SPRequestDuration":["17"],"SPIisLatency":["0"],"Server":["Microsoft-IIS/8.5"],"WWW-Authenticate":["Bearer
realm=\"97b27fa4-494a-4bff-a6a0-00b798335a86\",client_id=\"00000003-0000-0ff1-ce00-000000000000\",trusted_issuers=\"00000005-0000-0000-c000-000000000000#*,00000003-0000-0ff1-ce00-000000000000#97b27fa4-494a-4bff-a6a0-00b798335a86\"","Negotiate","NTLM"],"X-Powered-By":["ASP.NET"],"MicrosoftSharePointTeamServices":["15.0.0.4687"],"X-Content-Type-Options":["nosniff"],"X-MS-InvokeApp":["1;
RequireReadOnly"],"Date":["Mon, 14 Jan 2019 13:47:12 GMT"]} at
Microsoft.Activities.Hosting.Runtime.Subroutine.SubroutineChild.Execute(CodeActivityContext
context) at
System.Activities.CodeActivity.InternalExecute(ActivityInstance
instance, ActivityExecutor executor, BookmarkManager bookmarkManager)
at
System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor
executor, BookmarkManager bookmarkManager, Location resultLocation)
Tried many given solutions (User Sync (however manual start with the same user is not a problem)), Workflow App Permissions feature ..
I am logging the connection status events with the wcf relay, and I'm seeing something like this in the logs.
1/26 06:47:12 ERROR Service Bus ConnectionStatus: 'Reconnecting' Event. [(null)][42]
LastError: System.ServiceModel.CommunicationException: Exception of type 'System.ServiceModel.CommunicationException' was thrown. ---> Microsoft.ServiceBus.Messaging.Amqp.AmqpException: An AMQP error occurred (condition='amqp:unauthorized-access').
--- End of inner exception stack trace ---
This exception doesn't show up in the list on this microsoft page, and the only other post I can find anywhere related to this error message is here. However, that post does not have any recent comments or a resolution or workaround for the issue. Also, the exception doesn't have any stacktrace, so how am I supposed to troubleshoot this error?
I guess as a follow-up, I would ask whether this is anything to actually worry about if the wcf connection is never faulting.
Apparently, the token that the relay keeps refreshing to stay active requires the time on the server to match the azure service that it is connecting with, and if not, this type of error will show up. We were able to fix it by correcting the server time.
I've created a Windows Service (using Topshelf package) that uses WebApp.Start to listen to http requests. My code is as follows:
var options = new StartOptions();
options.Urls.Add( "http://localhost:9095" );
options.Urls.Add( "http://127.0.0.1:9095" );
options.Urls.Add( $"http://{ConfigurationHelper.IPAddress}:9095" );
options.Urls.Add( $"http://{Environment.MachineName}:9095" );
Microsoft.Owin.Hosting.WebApp.Start<Startup>( options );
I can hit all the desired URLs when I'm browsing on the Windows Service machine, but when I'm on a remote machine on the same network with access to the machine, I get a 500 error when hitting the URL (i.e. http://192.168.100.14:9095/hangfire/).
Some notes:
Windows Firewall is turned off, but I'm still confirming with IT whether or not our hardware firewall might be blocking something (would we get 500 if that was case or 404?)
The windows service is running as a local user that is part of the Administrators group.
For each of those URLs, I ran the netsh command: netsh http add urlacl url=http://127.0.0.1:9095/ user=myAdminUser
Any other ideas on why I'd get a 500 or how to see any diagnostic information anywhere? (debug view and event viewer have nothing).
It is my first Owin application, so I'm very green on it yet.
Update: OWIN error trace
[8496] BTR.Evolution.Service OWIN Exception: Object reference not set to an instance of an object.
[8496]
[8496] at Hangfire.Dashboard.MiddlewareExtensions.Unauthorized(IOwinContext owinContext)
[8496] at Hangfire.Dashboard.MiddlewareExtensions.<>c__DisplayClass1_2.b__1(IDictionary`2 env)
[8496] at Microsoft.Owin.Mapping.MapMiddleware.d__0.MoveNext()
[8496] --- End of stack trace from previous location where exception was thrown ---
[8496] at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
[8496] at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
[8496] at BTR.Evolution.Service.GlobalExceptionMiddleware.d__1.MoveNext() in C:\BTR\Source\Evolution.Service\BTR.Evolution.Service\Startup.cs:line 20
So maybe it wasn't OWIN that was the culprit but rather Hangfire and my attempt to use it remotely. Going to look into authorization filter to see if that will help.
Look into how to get the stack trace from the exception by creating an OWIN middleware logger. See Unhandled Exception Global Handler for OWIN / Katana? for more details.
Judging by the update that includes the stack trace you need to look into auth issues within hangfire. Check out http://docs.hangfire.io/en/latest/configuration/using-dashboard.html#configuring-authorization - This states that for security reasons, by default only local request are allowed. It also provides sample code showing how to use IAuthorizationFilter to allow remote request.
Sorry if the Title is a little vague but the truth is I do not understand my problem, I have never come across this before and I havent done anything out of the Ordinary.
Right I have a basic Silverlight 4 Application using the Business Application Template. I have also created a WCF Service Application Project which is used to simply return an Observable collection for some entries in my Database,
I published this to my Local computer which I then referenced in my SL4 app and I get this exception
[System.ServiceModel.CommunicationException: The remote server returned an error: NotFound. ---> System.Net.WebException: The remote server returned an error: NotFound. ---> System.Net.WebException: The remote server returned an error: NotFound.
at System.Net.Browser.BrowserHttpWebRequest.InternalEndGetResponse(IAsyncResult asyncResult)
at System.Net.Browser.BrowserHttpWebRequest.<>c__DisplayClass5. <EndGetResponse>b__4(Object sendState)
at System.Net.Browser.AsyncHelper.<>c__DisplayClass4.<BeginOnUI>b__0(Object sendState)
--- End of inner exception stack trace ---
at System.Net.Browser.AsyncHelper.BeginOnUI(SendOrPostCallback beginMethod, Object state)
at System.Net.Browser.BrowserHttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
at System.ServiceModel.Channels.HttpChannelFactory.HttpRequestChannel.HttpChannelAsyncRequest. CompleteGetResponse(IAsyncResult result)
--- End of inner exception stack trace ---
at System.ServiceModel.AsyncResult.End[TAsyncResult](IAsyncResult result)
at System.ServiceModel.Channels.ServiceChannel.EndCall(String action, Object[] outs, IAsyncResult result)
at System.ServiceModel.ClientBase`1.ChannelBase`1.EndInvoke(String methodName, Object[] args, IAsyncResult result)
at RMServiceRIA.rService.Service1Client.Service1ClientChannel.EndgetBuildings(IAsyncResult result)
at RMServiceRIA.rService.Service1Client.RMServiceRIA.rService.IService1.EndgetBuildings(IAsync Result result)
at RMServiceRIA.rService.Service1Client.OnEndgetBuildings(IAsyncResult result)
at System.ServiceModel.ClientBase`1.OnAsyncCallCompleted(IAsyncResult result)}
I have got a clientaccess policy and crossdomain policy which for the purpose of testing is set to allow anything access,
Client Access Policy
Cross Domain
any ideas on how to fix this problem, I debugged my WCF Service and it receives the call, the method executed reaches its return statement, yet my SL4 never even receives the result, as soon as my WCF service has the result I get an exception thrown within my SL4 app, to me it just seems that the connection gets closed but I dont know and I have tried to research but where do you start with something you dont fully understand? I found many people with the same sort of error message but WCF is so damn cryptic... I did try and enable the debugging information inside of my webconfig. I have also tried to host the WCF service on my Web server but no luck, I have several WCF Services running on that with a similar set up so I am unsure why this isnt working
Web Config for WCF Service
client config
Thanks for having a look, and if you need any more info just ask I'll be happy to provide you with it,
Your best bet is to turn on WCF Tracing so you can see what is going on.
http://software.intel.com/en-us/blogs/2010/02/03/adventures-with-silverlight-and-wcf-the-remote-server-returned-an-error-notfound/
Most likely there is an issue with serializing something. Not found is just the Silverlight version of a 500 error.