We Keep Coding

RabbitMQ Errors AlreadyClosedException

I have a .Net 6 microservice application which is receiving occasional RabbitMQ errors although there doesn't appear to be an excessive rate of messages on the queue it is trying to write to.
The error returned looks like
RabbitMQ.Client.Exceptions.AlreadyClosedException: Already closed: The
AMQP operation was interrupted: AMQP close-reason, initiated by
Library, code=541, text='Unexpected Exception', classId=0, methodId=0,
cause=System.IO.IOException: Unable to read data from the transport
connection: Connection reset by peer.\n --->
System.Net.Sockets.SocketException (104): Connection reset by peer\n
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset,
Int32 count)\n --- End of inner exception stack trace ---\n at
RabbitMQ.Client.Impl.InboundFrame.ReadFrom(NetworkBinaryReader
reader)\n at
RabbitMQ.Client.Framing.Impl.Connection.MainLoopIteration()\n at
RabbitMQ.Client.Framing.Impl.Connection.MainLoop()\n at
RabbitMQ.Client.Framing.Impl.Connection.EnsureIsOpen()\n at
RabbitMQ.Client.Framing.Impl.AutorecoveringConnection.CreateModel()\n
at ServiceStack.RabbitMq.RabbitMqExtensions.OpenChannel(IConnection
connection) in
/home/runner/work/ServiceStack/ServiceStack/ServiceStack/src/ServiceStack.RabbitMq/RabbitMqExtensions.cs:line
18\n at ServiceStack.RabbitMq.RabbitMqProducer.get_Channel() in
/home/runner/work/ServiceStack/ServiceStack/ServiceStack/src/ServiceStack.RabbitMq/RabbitMqProducer.cs:line
47\n at ServiceStack.RabbitMq.RabbitMqProducer.Publish(String
queueName, IMessage message, String exchange) in
/home/runner/work/ServiceStack/ServiceStack/ServiceStack/src/ServiceStack.RabbitMq/RabbitMqProducer.cs:line
99\n at
ASM.Helios.ServiceHosting.RabbitMqServiceRequestLogger.Log(IRequest
request, Object requestDto, Object response, TimeSpan
requestDuration)\n at
ServiceStack.Host.ServiceRunner`1.LogRequest(IRequest req, Object
requestDto, Object response) in
/home/runner/work/ServiceStack/ServiceStack/ServiceStack/src/ServiceStack/Host/ServiceRunner.cs:line
233
We have found that increasing the number of service instances does seem to help reduce the frequency of the errors, but they will still occur.
I was wondering if it is a similar issue to this stackoverflow question, in which case maybe setting the prefetch count to a lower value would help.
The code for setting up our rabbitMQ connection using the serviceStack implementation of rabbit looks like:
private static void SetUpRabbitMqConnection(IServiceDiscovery serviceDiscovery)
{
MessageService = RabbitMqServerFactory.GetRabbitMqServer(serviceDiscovery).Result;
MessageService.ConnectionFactory.SocketReadTimeout = 1000;
MessageService.ConnectionFactory.SocketWriteTimeout = 1000;
MessageService.ConnectionFactory.RequestedHeartbeat = 3;
MessageService.RetryCount = 0;
MqClient = (RabbitMqQueueClient)MessageService.MessageFactory.CreateMessageQueueClient();
ResponseQueueName = MqClient.GetTempQueueName(); //This creates a temp queue which gets auto deleted when nothing is connected to it
RabbitMqConsumer = new EventingBasicConsumer(MqClient.Channel);
MqClient.Channel.BasicConsume(queue: ResponseQueueName, consumer: RabbitMqConsumer, noLocal: true);
Console.WriteLine(" [x] Awaiting RPC requests");
RabbitMqConsumer.Received -= RabbitMqConsumerOnReceived;
RabbitMqConsumer.Received += RabbitMqConsumerOnReceived;
Disconnected = false;
}
Would adding a line like: MqClient.Channel.BasicQos(prefetchSize, prefetchCount,global); help?
What are sensible values for the 3 parameters? I think the defaults are 0, 20, and false.
Or is there a different configuration change that might help?

WebSocketSharp cannot catch exceptions

I am having trouble catching and handling exceptions thrown by websocketsharp
When I connect to the server with the client, and I close the client just by ALT+F4 or X button, the server receives the exception of connection forcibly closed.
Even though, these exceptions won't crash the program, this will cause the console to be spammed by those exceptions, important messages and logs will be pushed down and nothing can be read (there is a lot of clients).
This is the exception:
7/28/2021 6:41:56 AM|Fatal|<>c__DisplayClass71_0.<receiveRequest>b__0:0|WebSocketSharp.WebSocketException: An exception has occurred while reading an HTTP request/response.
---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..
---> System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.
--- End of inner exception stack trace ---
at System.Net.Sockets.NetworkStream.Read(Span`1 buffer)
at System.Net.Sockets.NetworkStream.ReadByte()
at WebSocketSharp.HttpBase.readHeaders(Stream stream, Int32 maxLength)
at WebSocketSharp.HttpBase.Read[T](Stream stream, Func`2 parser, Int32 millisecondsTimeout)
--- End of inner exception stack trace ---
at WebSocketSharp.HttpBase.Read[T](Stream stream, Func`2 parser, Int32 millisecondsTimeout)
at WebSocketSharp.HttpRequest.Read(Stream stream, Int32 millisecondsTimeout)
at WebSocketSharp.Net.WebSockets.TcpListenerWebSocketContext..ctor(TcpClient tcpClient, String protocol, Boolean secure, ServerSslConfiguration sslConfig, Logger logger)
at WebSocketSharp.Ext.GetWebSocketContext(TcpClient tcpClient, String protocol, Boolean secure, ServerSslConfiguration sslConfig, Logger logger)
at WebSocketSharp.Server.WebSocketServer.<>c__DisplayClass71_0.<receiveRequest>b__0(Object state)
I tried using
public class Echo : WebSocketBehavior
{
protected override void OnError(ErrorEventArgs e)
{
//handle
}
I tried wrapping the websocket server in a try catch
try
{
string path = "ws://localhost:111";
WebSocketServer wssv = new WebSocketServer(path);
wssv.Start();
while (Console.ReadKey(true).Key != ConsoleKey.Enter) { }
wssv.Stop();
} catch { }
Nothing works, the exception will the thrown and shown on the console whatever I tried.
So the question is, how do you catch those exceptions?

I was able to solve the issue disabling/hiding completely the logs output.
public static void Disable(this Logger logger)
{
var field = logger.GetType().GetField("_output", BindingFlags.NonPublic | BindingFlags.Instance);
field?.SetValue(logger, new Action<LogData, string>((d, s) => { }));
}

System.ObjectDisposedException throwed on WebSocket communication

I'm trying to communicate between a web browser client and an ASP.NET server using WebSockets.
I make a set of requests, of different sizes and with some seconds of elapsed time between each of them. The three first ones pass correctly, but a precise one, with nothing in particular from the other, close the WebSocket connection, throw an exception on server side.
The error message and stack trace of this exception look like this :
FATAL ERROR: Cannot access a disposed object.
Object name: 'System.Web.WebSockets.AspNetWebSocket'.
at System.Web.WebSockets.AspNetWebSocket.ThrowIfDisposed()
at System.Web.WebSockets.AspNetWebSocket.SendAsyncImpl(ArraySegment 1 buffer, WebSocketMessageType messageType, Boolean endOfMessage, CancellationToken cancellationToken, Boolean performValidation)
at System.Web.WebSockets.AspNetWebSocket.SendAsync(ArraySegment 1 buffer, WebSocketMessageType messageType, Boolean endOfMessage, CancellationToken cancellationToken)
at [my code path here...]
It may be a threading problem, because I'm using async methods everywhere from functions that communicate with websockets.
I know the exception is throwed from this code (at socket.SendAsync):
public class SocketTranslater
{
private WebSocket socket;
private JavaScriptSerializer serializer;
// [...]
private ArraySegment<byte> Encode(Object data)
{
string json = serializer.Serialize(data);
return (new ArraySegment<byte>(Encoding.UTF8.GetBytes(json)));
}
public async Task Send(Object packet)
{
ArraySegment<byte> encoded = this.Encode(packet);
await socket.SendAsync(encoded, WebSocketMessageType.Text, true, CancellationToken.None);
}
}
The socket is created from another class :
public class EventSender : IHttpHandler
{
private static List<SocketTranslater> socketTranslaters = new List<SocketTranslater>();
public void ProcessRequest(HttpContext context)
{
this.id = context.Request.UserHostName + ":" + context.Request.UserAgent;
if (context.IsWebSocketRequest)
{
context.AcceptWebSocketRequest(ProcessSocket);
}
}
private async Task ManageSocket(AspNetWebSocketContext context)
{
WebSocket socket = context.WebSocket;
SocketTranslater translater = new SocketTranslater(socket);
translaters.add(translater);
while (true)
{
// Manage messages, using the other class to send responses.
translater.Send(/* Any struct here */);
}
}
Unhopefully, my project is too big to put all the code here.
Any idea of error source, or additional information that you would require ?
UPDATE:
After some more tests, I don't have this exception from time to time. I always do the same thing, but the server seems to have a random comportment.
That makes my problem even weirder...

Finally, after some more tests and interesting questions and answer from here (like this one), I understood:
My problem was that I was stocking WebSockets in a Dictionary linked with hostnames. So on the first connection of a client on my server, everything worked correctly. But if I refresh the page, the websocket was closed by the server (because there was no chance to use it again) et another one was created.
But as I used the same key for both sockets, the deprecated and the new one, I was trying to answer to the new client with the previous socket, that was closed. (disposed = closed for ASP.NET).
So the only thing that I had to do is to remove a socket from the list on the client disconnection (at the end of ManageSocket). And forbid a client to connect twice with the same hostname.
I didn't mention the part where I was linking sockets with hostnames, so I admit you couldn't really help me... I apologize.

No matter what I try: The I/O operation has been aborted because of either a thread exit or an application request

I try to build a simple async net tcp wcf tool which will open connection, send command, receive answer (a List with 0-10 string sentences), close connection.
The problem is, I get on (self-hosted) service side always - no matter what I try - "The I/O operation has been aborted because of either a thread exit or an application request", on client side of course the corresponding errors like "Existing connection was closed by remote host" and timeouts and so on.
I tried alot for the past days but I can't get rid of it.
Client Side (running on .NET 4.0, called around once a sec):
void callservice(string mykey) {
ServiceReference1.Client c = new ServiceReference1.Client();
c.GetDataCompleted += c_GetDataCompleted;
try {
c.GetDataAsync(mykey);
}
catch (FaultException aa)
{
c.Abort();
}
}
private void c_GetDataCompleted(object sender, ServiceReference1.GetDataCompletedEventArgs e)
{
ServiceReference1.Client c = (ServiceReference1.Client)sender;
c.GetDataCompleted -= c_GetDataCompleted;
try
{
if (e.Result != null && e.Result.Length > 0)
{
... }
c.Close();
}
catch (Exception) {
c.Abort();
}
}
Server Side (running on .NET4.5):
[ServiceBehavior(ConcurrencyMode=ConcurrencyMode.Multiple,
InstanceContextMode=InstanceContextMode.PerCall,IncludeExceptionDetailInFaults=true)]
public class Service1 : IMyService
{
public async Task<List<string>> GetData(string whatkey)
{
List<string> mydatalist = new List<string>();
mydatalist= await Task.Run<List<string>>(() =>
{
...
});
return mydatalist;
}
What is going wrong there? Could it be that it is something not having to do with WCF at all? What could it be?
Server Side Exception:
System.Net.Sockets.SocketException, System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089
The I/O operation has been aborted because of either a thread exit or an application request
at System.ServiceModel.Channels.SocketConnection.HandleReceiveAsyncCompleted()
at System.ServiceModel.Channels.SocketConnection.OnReceiveAsync(Object sender, SocketAsyncEventArgs eventArgs)
at System.Net.Sockets.SocketAsyncEventArgs.FinishOperationAsyncFailure(SocketError socketError, Int32 bytesTransferred, SocketFlags flags)
at System.Net.Sockets.SocketAsyncEventArgs.CompletionPortCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped)
at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)
System.Net.Sockets.SocketException (0x80004005): The I/O operation has been aborted because of either a thread exit or an application request
3E3
One more interesting fact:
SVCLogs show me that the I/O Exeption occurs after a timespan I can define in the
<connectionPoolSettings groupName="default" leaseTimeout="00:03:00"
idleTimeout="00:02:39" maxOutboundConnectionsPerEndpoint="20" />
settings.
In this example it will occur the first time after 00:02:39.
My interpretation: It closes open connections due to the settings there and that causes the Exception since the ReceiveAsync operation i.ex. was still open.
My question is so far why does client.close() not close it completely and why isn't it finished yet when it is calling the c_getdatacompleted-event? Why does the operation "hang out" for 02:39 minutes and does not come to an end?
(If I would not force the close down via the connectionpool settings I end up with hundreds of open operations if I use netstat i.ex. to display)

Async WCF operations (AsyncPattern=true) are implemented with the Asynchronous Programming Model. That is, you implement an operation ("Operation") with two asynchronous operations ("BeginOperation" and "EndOeration"). The client can wrap those operations with a Task (presumably with the FromAsync overload)
For example:
[ServiceContract]
public interface ISampleTaskAsync
{
[OperationContract(AsyncPattern = true)]
IAsyncResult BeginDoWork(int count, AsyncCallback callback, object state);
int EndDoWork(IAsyncResult result);
}
The WCF contract does not return a Task<T>
Then, on the client you could do something like:
var proxy = new Services.SampleTaskAsyncClient();
object state = "This can be whatever you want it to be";
var task = Task<int>.Factory.FromAsync(proxy.BeginDoWork,
proxy.EndDoWork, 10, state);
For more information see:
http://msdn.microsoft.com/en-us/library/system.servicemodel.operationcontractattribute.asyncpattern.aspx
http://blogs.msdn.com/b/rjacobs/archive/2011/06/30/how-to-implement-a-wcf-asynchronous-service-operation-with-task-lt-t-gt.aspx
If you want to use Task<T>, I believe you don't need AsyncPattern=true.

MSMQ Access issue in NServiceBus with asp.net web service

I am trying to implement publisher - subscribe in my project of asp.net (wcf) web services. When i am trying to create bus in global.asax
protected void Application_Start(object sender, EventArgs e)
{
try
{
log4net.Config.XmlConfigurator.Configure();
Bus = NServiceBus.Configure.WithWeb()
.SpringBuilder()
.BinarySerializer()
.MsmqTransport()
.IsTransactional(false)
.PurgeOnStartup(false)
.UnicastBus()
.ImpersonateSender(false)
.CreateBus()
.Start();
}
catch (Exception ex)
{
log.Error(ex);
}
}
public static IBus Bus { get; private set; }
but it is not creating bus.. then it is throwing exception
Exception reached top level.
System.Messaging.MessageQueueException: Access to Message Queuing system is denied.
at System.Messaging.MessageQueue.MQCacheableInfo.get_ReadHandle()
at System.Messaging.MessageQueue.StaleSafeReceiveMessage(UInt32 timeout, Int32 action, MQPROPS properties, NativeOverlapped* overlapped, ReceiveCallback receiveCallback, CursorHandle cursorHandle, IntPtr transaction)
at System.Messaging.MessageQueue.ReceiveCurrent(TimeSpan timeout, Int32 action, CursorHandle cursor, MessagePropertyFilter filter, MessageQueueTransaction internalTransaction, MessageQueueTransactionType transactionType)
at System.Messaging.MessageQueue.Peek(TimeSpan timeout)
at NServiceBus.Unicast.Transport.Msmq.MsmqTransport.Receive()
at NServiceBus.Utils.WorkerThread.Loop()
How Can I Avoid this?.... .... I am struck in the middle...
nRk

This is most likely a security issue, make sure to give everyone full rights to the queues involved
hope this helps!