We Keep Coding

Unable to stop asp.net core Windows Service when Kestrel receives requests

We are using asp.net core running on .net 4.5.1 as Windows Service. Kestrel is used as web server (Kestrel version is 1.1.2).
It was configured in accordance with Host an ASP.NET Core app in a Windows Service. We also using SignalR running on OWIN.
The problem occurs when two events occur simultaneously: Windows Service is stopping and some new web requests were received by Krestel. If these conditions are met, then Windows service stops responding and hangs.
Windows could not stop the Topshelf.Host service on Local Computer.
Error 1061: The service cannot accept control messages at this time.
Confuguration code:
var port = Global.Settings.Port;
var hostBuilder = new WebHostBuilder()
.UseKestrel()
.UseStartup<Startup>()
.UseUrls("http://+:" + port);
if (runAsService)
{
hostBuilder.UseContentRoot(directoryPath);
Directory.SetCurrentDirectory(directoryPath);
}
else
{
hostBuilder.UseContentRoot(Directory.GetCurrentDirectory());
}
var host = hostBuilder.Build();
if (runAsService)
{
host.RunAsCustomService();
}
else
{
host.Run();
}
public static class WebHostServiceExtensions
{
public static void RunAsCustomService(this IWebHost host)
{
var webHostService = new CustomWebHostService(host);
ServiceBase.Run(webHostService);
}
}
internal class CustomWebHostService : WebHostService
{
public CustomWebHostService(IWebHost host) : base(host)
{
}
//...
}
Further investigation revealed that unhandled exception occured when disposing _host in WebHostService.cs.
protected sealed override void OnStop()
{
_stopRequestedByWindows = true;
OnStopping();
_host?.Dispose();
OnStopped();
}
As described below, it raises an exception "Microsoft.AspNetCore.Server.Kestrel.Internal.Networking.UvException: Error -4082 EBUSY resource busy or locked"
The Question is:
Is there a way to avoid such an error and prevent Windows Service hangs? E.g. stop Krestel before _host disposing or prevent new incoming connection in OnStopping event.
Stacktrace:
Microsoft.AspNetCore.Server.Kestrel.Internal.Networking.UvException:
Error -4082 EBUSY resource busy or locked
in Microsoft.AspNetCore.Server.Kestrel.Internal.Networking.Libuv.loop_close(UvLoopHandle
handle) in
Microsoft.AspNetCore.Server.Kestrel.Internal.Networking.UvLoopHandle.ReleaseHandle()
in System.Runtime.InteropServices.SafeHandle.InternalDispose() in
System.Runtime.InteropServices.SafeHandle.Dispose(Boolean disposing)
in System.Runtime.InteropServices.SafeHandle.Dispose() in
Microsoft.AspNetCore.Server.Kestrel.Internal.KestrelThread.ThreadStart(Object
parameter)
in
Microsoft.AspNetCore.Server.Kestrel.Internal.KestrelThread.d__51.MoveNext()
in System.Threading.Tasks.Task.WaitAll(Task[] tasks, Int32
millisecondsTimeout, CancellationToken cancellationToken) in
Microsoft.AspNetCore.Server.Kestrel.Internal.KestrelEngine.Dispose()
in Microsoft.AspNetCore.Server.Kestrel.KestrelServer.Dispose() in
Microsoft.Extensions.DependencyInjection.ServiceProvider.Dispose()
in Microsoft.AspNetCore.Hosting.Internal.WebHost.Dispose()
Microsoft.AspNetCore.Server.Kestrel.Internal.Networking.UvException:
Error -4082 EBUSY resource busy or locked in
Microsoft.AspNetCore.Server.Kestrel.Internal.Networking.Libuv.loop_close(UvLoopHandle
handle) in
Microsoft.AspNetCore.Server.Kestrel.Internal.Networking.UvLoopHandle.ReleaseHandle()
in System.Runtime.InteropServices.SafeHandle.InternalDispose() in
System.Runtime.InteropServices.SafeHandle.Dispose(Boolean disposing)
in System.Runtime.InteropServices.SafeHandle.Dispose() in
Microsoft.AspNetCore.Server.Kestrel.Internal.KestrelThread.ThreadStart(Object
parameter)
в
Microsoft.AspNetCore.Server.Kestrel.Internal.KestrelThread.d__51.MoveNext()

The EBUSY exceptions thrown from KestrelServer.Dispose() will be fixed in the 2.0 release of Kestrel. I think the reason your service is hanging is because _host?.Dispose() is throwing which prevents OnStopped(); from being called inside the OnStop() method.
In the meantime, swallowing the exception to ensure OnStopped(); always gets called should allow the service to be stopped. Make sure to build a new WebHost if you want to restart the server.

RabbiMQ C# driver Causes System.AccessViolationException

We have an ASP.Net WebAPI 2.0 application that publishes messages to our RabbitMQ Server. In 99% of the cases, all is well... but randomly the application terminates with a System.AccessViolationException for no apparent reason.
How can I prevent this failure? Could it be related to our recent upgrade to the 3.6.6 C# driver (it worked fine before the upgrade)?
Things I have already eliminated:
A new IModel is used for each publish (I know IModel is not thread
safe)
A call is made to CreateConnection for each call as well (I
know I could reuse the connection, but we do not currently). The connection is AutoClose = true;
The Channel is used in a using block... so it is disposed each time
Here is a sample stack trace of where it explodes:
Exception Details
System.AccessViolationException
Attempted to read or write protected
memory. This is often an indication that other memory is corrupt.
at System.Net.UnsafeNclNativeMethods.SafeNetHandlesXPOrLater.GetAddrInfoW(String
nodename, String servicename, AddressInfo& hints, SafeFreeAddrInfo&
handle)
at System.Net.Dns.TryGetAddrInfo(String name, AddressInfoHints flags, IPHostEntry& hostinfo)
at System.Net.Dns.InternalGetHostByName(String hostName, Boolean includeIPv6)
at System.Net.Dns.GetHostAddresses(String hostNameOrAddress)
at RabbitMQ.Client.TcpClientAdapter.BeginConnect(String host, Int32 port, AsyncCallback requestCallback, Object state)
at RabbitMQ.Client.Impl.SocketFrameHandler.Connect(ITcpClient socket, AmqpTcpEndpoint endpoint, Int32 timeout)
at RabbitMQ.Client.Impl.SocketFrameHandler..ctor(AmqpTcpEndpoint endpoint, Func`2 socketFactory, Int32 connectionTimeout, Int32
readTimeout, Int32 writeTimeout)
at RabbitMQ.Client.Framing.Impl.ProtocolBase.CreateFrameHandler(AmqpTcpEndpoint
endpoint, Func'2 socketFactory, Int32 connectionTimeout, Int32
readTimeout, Int32 writeTimeout)
at RabbitMQ.Client.ConnectionFactory.CreateConnection(IList'1 endpoints, String clientProvidedName)
And another
System.Net.UnsafeNclNativeMethods+SafeNetHandlesXPOrLater.GetAddrInfoW(System.String,
System.String, System.Net.AddressInfo ByRef,
System.Net.SafeFreeAddrInfo ByRef)
System.Net.Dns.TryGetAddrInfo(System.String, System.Net.AddressInfoHints, System.Net.IPHostEntry ByRef)
System.Net.Dns.InternalGetHostByName(System.String, Boolean)
System.Net.Dns.GetHostAddresses(System.String)
RabbitMQ.Client.TcpClientAdapter.BeginConnect(System.String, Int32, System.AsyncCallback, System.Object)
RabbitMQ.Client.Impl.SocketFrameHandler.Connect(RabbitMQ.Client.ITcpClient,
RabbitMQ.Client.AmqpTcpEndpoint, Int32)
RabbitMQ.Client.Impl.SocketFrameHandler..ctor(RabbitMQ.Client.AmqpTcpEndpoint,
System.Func'2,
Int32, Int32, Int32)
RabbitMQ.Client.Framing.Impl.ProtocolBase.CreateFrameHandler(RabbitMQ.Client.AmqpTcpEndpoint,
System.Func'2,
Int32, Int32, Int32)
RabbitMQ.Client.ConnectionFactory.CreateConnection(System.Collections.Generic.IList'1,
System.String)

I'm not sure exactly why your error is occurring I would need to see some of your code for that, but I use RabbitMQ quite a bit and to publish I use a class like so:
(Changed some parts as they are not relevant for your situation such as encryption, compression etc. But this would be the basic format of it)
using System;
using System.Text;
using RabbitMQ.Client;
using RabbitMQ.Client.Framing;
namespace Messaging
{
public class MessageSender : IDisposable
{
private const string EXCHANGE_NAME = "MY_EXCHANGE";
private readonly ConnectionFactory factory;
private readonly IConnection connection;
private readonly IModel channel;
public MessageSender(string address, string username, string password)
{
factory = new ConnectionFactory {UserName = username, Password = password, HostName = address};
connection = factory.CreateConnection();
channel = connection.CreateModel();
channel.ExchangeDeclare(EXCHANGE_NAME, "topic");
}
public void Send(string payload, string topic)
{
var prop = new BasicProperties();
var data = Encoding.ASCII.GetBytes(payload);
channel.BasicPublish(EXCHANGE_NAME, topic.ToUpper(), prop, data);
}
public void Dispose()
{
try
{
channel.Dispose();
connection.Dispose();
}
catch (Exception e)
{
Console.WriteLine(e);
}
}
}
}
The idea is to let you issue multiple calls or a single one and dispose of the class when you wish. Wrap it with a using statement and your set.
Never had a single problem with this in about 3-4 years of using it, so you might be able to contrast this vs your code to find the differences.

Managed to get a (partial) resolution on this from RabbitMQ users group.
Details here: Rabbit MQ Users Group
Basic idea is that the IConnection object should be shared, as it is heavy to open and close. Only the IModel should be opened fresh for each thread

Event log write error

It is simple, I want to write something to event log.
protected override void OnStop()
{
// TODO: Add code here to perform any tear-down necessary to stop your service.
if (!System.Diagnostics.EventLog.SourceExists("IvrService"))
{
System.Diagnostics.EventLog.CreateEventSource(
"IvrService", "IvrServiceLog");
}
EventLog eventLog1 = new System.Diagnostics.EventLog();
eventLog1.Source = "IvrService";
eventLog1.Log = "IvrServiceLog";
try
{
eventLog1.WriteEntry("Successfully "+State.Stopped.ToString());
IvrApplication.StopImmediate();
}
catch (Exception ex)
{
// eventLog1.WriteEntry(ex.Message);
}
}
The exception is:
Failed to stop service. System.ArgumentException: The source 'IvrService' is not registered in log 'IvrServiceLog'. (It is registered in log 'Application'.) " The Source and Log properties must be matched, or you may set Log to the empty string, and it will automatically be matched to the Source property.
at System.Diagnostics.EventLogInternal.VerifyAndCreateSource(String sourceName, String currentMachineName)
at System.Diagnostics.EventLogInternal.WriteEntry(String message, EventLogEntryType type, Int32 eventID, Int16 category, Byte[] rawData)
at System.Diagnostics.EventLog.WriteEntry(String message)

The error message is telling you exactly what is wrong. You have the Event Source IvrService registered with the Application Log, not the IvrServiceLog. The System.Diagnostics.EventLog.SourceExists verifies that the source exists, but not for a particular log.
My guess is that you originally registered this with the Application log and then later changed it to write to the IvrServiceLog.
To clean up your development machine, you could simply run the following and then you code should work going forward.
System.Diagnostics.EventLog.DeleteEventSource("IvrService");

No matter what I try: The I/O operation has been aborted because of either a thread exit or an application request

I try to build a simple async net tcp wcf tool which will open connection, send command, receive answer (a List with 0-10 string sentences), close connection.
The problem is, I get on (self-hosted) service side always - no matter what I try - "The I/O operation has been aborted because of either a thread exit or an application request", on client side of course the corresponding errors like "Existing connection was closed by remote host" and timeouts and so on.
I tried alot for the past days but I can't get rid of it.
Client Side (running on .NET 4.0, called around once a sec):
void callservice(string mykey) {
ServiceReference1.Client c = new ServiceReference1.Client();
c.GetDataCompleted += c_GetDataCompleted;
try {
c.GetDataAsync(mykey);
}
catch (FaultException aa)
{
c.Abort();
}
}
private void c_GetDataCompleted(object sender, ServiceReference1.GetDataCompletedEventArgs e)
{
ServiceReference1.Client c = (ServiceReference1.Client)sender;
c.GetDataCompleted -= c_GetDataCompleted;
try
{
if (e.Result != null && e.Result.Length > 0)
{
... }
c.Close();
}
catch (Exception) {
c.Abort();
}
}
Server Side (running on .NET4.5):
[ServiceBehavior(ConcurrencyMode=ConcurrencyMode.Multiple,
InstanceContextMode=InstanceContextMode.PerCall,IncludeExceptionDetailInFaults=true)]
public class Service1 : IMyService
{
public async Task<List<string>> GetData(string whatkey)
{
List<string> mydatalist = new List<string>();
mydatalist= await Task.Run<List<string>>(() =>
{
...
});
return mydatalist;
}
What is going wrong there? Could it be that it is something not having to do with WCF at all? What could it be?
Server Side Exception:
System.Net.Sockets.SocketException, System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089
The I/O operation has been aborted because of either a thread exit or an application request
at System.ServiceModel.Channels.SocketConnection.HandleReceiveAsyncCompleted()
at System.ServiceModel.Channels.SocketConnection.OnReceiveAsync(Object sender, SocketAsyncEventArgs eventArgs)
at System.Net.Sockets.SocketAsyncEventArgs.FinishOperationAsyncFailure(SocketError socketError, Int32 bytesTransferred, SocketFlags flags)
at System.Net.Sockets.SocketAsyncEventArgs.CompletionPortCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped)
at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)
System.Net.Sockets.SocketException (0x80004005): The I/O operation has been aborted because of either a thread exit or an application request
3E3
One more interesting fact:
SVCLogs show me that the I/O Exeption occurs after a timespan I can define in the
<connectionPoolSettings groupName="default" leaseTimeout="00:03:00"
idleTimeout="00:02:39" maxOutboundConnectionsPerEndpoint="20" />
settings.
In this example it will occur the first time after 00:02:39.
My interpretation: It closes open connections due to the settings there and that causes the Exception since the ReceiveAsync operation i.ex. was still open.
My question is so far why does client.close() not close it completely and why isn't it finished yet when it is calling the c_getdatacompleted-event? Why does the operation "hang out" for 02:39 minutes and does not come to an end?
(If I would not force the close down via the connectionpool settings I end up with hundreds of open operations if I use netstat i.ex. to display)

Async WCF operations (AsyncPattern=true) are implemented with the Asynchronous Programming Model. That is, you implement an operation ("Operation") with two asynchronous operations ("BeginOperation" and "EndOeration"). The client can wrap those operations with a Task (presumably with the FromAsync overload)
For example:
[ServiceContract]
public interface ISampleTaskAsync
{
[OperationContract(AsyncPattern = true)]
IAsyncResult BeginDoWork(int count, AsyncCallback callback, object state);
int EndDoWork(IAsyncResult result);
}
The WCF contract does not return a Task<T>
Then, on the client you could do something like:
var proxy = new Services.SampleTaskAsyncClient();
object state = "This can be whatever you want it to be";
var task = Task<int>.Factory.FromAsync(proxy.BeginDoWork,
proxy.EndDoWork, 10, state);
For more information see:
http://msdn.microsoft.com/en-us/library/system.servicemodel.operationcontractattribute.asyncpattern.aspx
http://blogs.msdn.com/b/rjacobs/archive/2011/06/30/how-to-implement-a-wcf-asynchronous-service-operation-with-task-lt-t-gt.aspx
If you want to use Task<T>, I believe you don't need AsyncPattern=true.

.Net Sockets -> throw System.InvalidOperationException

I've been having this issue for a while now and was never really able to resolve it. This issue only appears when I'm using the SendAsync/ReceiveAsync methods rather than the Begin/EndSend socket methods for asynchronous operations.
If have a fairly sophisticated tcp socket library and been meaning to replace the BeginSend methods with SendAsync but because of the issue I'm experiencing I always had to put it off. My socket server is handling heavy stress scenarios with > 1000 clients connected pushing constantly over 100mbit/sec and I'd like to utilize SendAsync method so I don't have the IAsyncResult allocation overhead.
Anyways, what happens is, everything works fine as long as I'm just sending/receiving data, however under high stress scenarios when the server is trying to disconnect/shutdown a client I'm occasionally getting the following Exception:
System.InvalidOperationException was unhandled
Message=Cannot apply a context that has been marshaled across AppDomains, that was not acquired through a Capture operation or that has already been the argument to a Set call.
Source=mscorlib
StackTrace:
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Net.Sockets.SocketAsyncEventArgs.FinishOperationSuccess(SocketError socketError, Int32 bytesTransferred, SocketFlags flags)
at System.Net.Sockets.SocketAsyncEventArgs.CompletionPortCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped)
at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)
InnerException:
I'm unable to catch this exception anywhere as it seems to happen within the .NET framework and I can't do anything about it crashing my server.
As this only happens when I call my shutdown procedure, I'm guessing it has something to do with calling Shutdown on the socket while it still has a Read/Write procedure pending.
However, I also tried to delay calling shutdown until all read/write Send/ReceiveAsync calls have returned and call shutdown after that but that didn't help either.
Here's how I try to shutdown sockets:
private void InternalDisconnect(SocketError socketError)
{
lock (shutdownLock)
{
if (isShutdown)
return;
isShutdown = true;
}
allowSend = false;
SocketError = socketError;
ThreadPool.QueueUserWorkItem(delegate
{
lock (padLock)
{
try
{
if (TcpSocketStatus == TcpSocketStatus.Disconnected)
return;
TcpSocketStatus = TcpSocketStatus.Disconnecting;
if (asyncSendArgs != null)
{
asyncSendArgs.Completed -= SendCompleted;
asyncSendArgs.SetBuffer(null, 0, 0);
asyncSendArgs.Dispose();
}
if (asyncReceiveArgs != null)
{
asyncReceiveArgs.Completed -= ReceiveCompleted;
asyncReceiveArgs.SetBuffer(null, 0, 0);
asyncReceiveArgs.Dispose();
}
try
{
bufferedSender.Clear();
Socket.Shutdown(SocketShutdown.Both);
if (Socket.Connected)
{
Socket.Disconnect(true);
}
}
catch
{
}
try
{
Socket.Close();
}
catch
{
}
TcpSocketStatus = TcpSocketStatus.Disconnected;
if (socketError != SocketError.Success)
{
if (log.IsDebugEnabled)
log.Debug("SocketDisconnected\tSocketError:{0}", socketError);
}
else
{
if (log.IsDebugEnabled)
log.Debug("SocketDisconnected");
}
DisconnectTime = DateTime.UtcNow;
if (TcpSocketDisconnected != null)
TcpSocketDisconnected(this);
}
catch (Exception ex)
{
log.ErrorException("InternalDisconnect", ex);
}
}
});
}

Ok, for some reason it always helps me to ask question because I tend to figure out things always shortly after :/
It seems that asyncSendArgs.SetBuffer(null, 0, 0); is the culprit.
Even though that another piece of the code should make sure that there are no outstanding async operations before InternalDisconnect is called it seems like there still are and altering the buffer while it's in use would definitely cause an issue.
I've been running heavy stress tests for 15 minutes so far and everything seems fine (before I always got that exception within one minute). I'll let it run for some more time and hope that that did it.