We Keep Coding

RabbitMQ Errors AlreadyClosedException

I have a .Net 6 microservice application which is receiving occasional RabbitMQ errors although there doesn't appear to be an excessive rate of messages on the queue it is trying to write to.
The error returned looks like
RabbitMQ.Client.Exceptions.AlreadyClosedException: Already closed: The
AMQP operation was interrupted: AMQP close-reason, initiated by
Library, code=541, text='Unexpected Exception', classId=0, methodId=0,
cause=System.IO.IOException: Unable to read data from the transport
connection: Connection reset by peer.\n --->
System.Net.Sockets.SocketException (104): Connection reset by peer\n
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset,
Int32 count)\n --- End of inner exception stack trace ---\n at
RabbitMQ.Client.Impl.InboundFrame.ReadFrom(NetworkBinaryReader
reader)\n at
RabbitMQ.Client.Framing.Impl.Connection.MainLoopIteration()\n at
RabbitMQ.Client.Framing.Impl.Connection.MainLoop()\n at
RabbitMQ.Client.Framing.Impl.Connection.EnsureIsOpen()\n at
RabbitMQ.Client.Framing.Impl.AutorecoveringConnection.CreateModel()\n
at ServiceStack.RabbitMq.RabbitMqExtensions.OpenChannel(IConnection
connection) in
/home/runner/work/ServiceStack/ServiceStack/ServiceStack/src/ServiceStack.RabbitMq/RabbitMqExtensions.cs:line
18\n at ServiceStack.RabbitMq.RabbitMqProducer.get_Channel() in
/home/runner/work/ServiceStack/ServiceStack/ServiceStack/src/ServiceStack.RabbitMq/RabbitMqProducer.cs:line
47\n at ServiceStack.RabbitMq.RabbitMqProducer.Publish(String
queueName, IMessage message, String exchange) in
/home/runner/work/ServiceStack/ServiceStack/ServiceStack/src/ServiceStack.RabbitMq/RabbitMqProducer.cs:line
99\n at
ASM.Helios.ServiceHosting.RabbitMqServiceRequestLogger.Log(IRequest
request, Object requestDto, Object response, TimeSpan
requestDuration)\n at
ServiceStack.Host.ServiceRunner`1.LogRequest(IRequest req, Object
requestDto, Object response) in
/home/runner/work/ServiceStack/ServiceStack/ServiceStack/src/ServiceStack/Host/ServiceRunner.cs:line
233
We have found that increasing the number of service instances does seem to help reduce the frequency of the errors, but they will still occur.
I was wondering if it is a similar issue to this stackoverflow question, in which case maybe setting the prefetch count to a lower value would help.
The code for setting up our rabbitMQ connection using the serviceStack implementation of rabbit looks like:
private static void SetUpRabbitMqConnection(IServiceDiscovery serviceDiscovery)
{
MessageService = RabbitMqServerFactory.GetRabbitMqServer(serviceDiscovery).Result;
MessageService.ConnectionFactory.SocketReadTimeout = 1000;
MessageService.ConnectionFactory.SocketWriteTimeout = 1000;
MessageService.ConnectionFactory.RequestedHeartbeat = 3;
MessageService.RetryCount = 0;
MqClient = (RabbitMqQueueClient)MessageService.MessageFactory.CreateMessageQueueClient();
ResponseQueueName = MqClient.GetTempQueueName(); //This creates a temp queue which gets auto deleted when nothing is connected to it
RabbitMqConsumer = new EventingBasicConsumer(MqClient.Channel);
MqClient.Channel.BasicConsume(queue: ResponseQueueName, consumer: RabbitMqConsumer, noLocal: true);
Console.WriteLine(" [x] Awaiting RPC requests");
RabbitMqConsumer.Received -= RabbitMqConsumerOnReceived;
RabbitMqConsumer.Received += RabbitMqConsumerOnReceived;
Disconnected = false;
}
Would adding a line like: MqClient.Channel.BasicQos(prefetchSize, prefetchCount,global); help?
What are sensible values for the 3 parameters? I think the defaults are 0, 20, and false.
Or is there a different configuration change that might help?

C# UDP An existing connection was forcibly closed by the remote host

I am creating a server for a game that handles multiple clients over UDP using the asynchronous methods, and am specifically working on clean disconnect logic. When a client hard crashes (their program is closed without proper disconnect logic) the readCallback on the server throws the SocketException
An existing connection was forcibly closed by the remote host
which makes sense, however when the read is triggered the next time on the loop in read it crashes despite the exception being handled in the callback.
private void connectedState()
{
while (connected)
{
//reset the trigger to non-signaled
readDone.Reset();
read(socket);
//block on reading data
readDone.WaitOne();
}
}
private void read(Socket sock)
{
// Creates an IpEndPoint to capture the identity of the sending host.
IPEndPoint sender = new IPEndPoint(IPAddress.Any, 0);
EndPoint senderRemote = sender;
// Create the state object.
StateObject state = new StateObject();
state.workSocket = sock;
//crashes after an exception is caught within the callback
sock.BeginReceiveFrom(state.buffer, 0, StateObject.MESSAGE_SIZE, SocketFlags.None, ref senderRemote, new AsyncCallback(readCallback), state);
}
private void readCallback(IAsyncResult ar)
{
StateObject state = (StateObject)ar.AsyncState;
Socket sock = state.workSocket;
EndPoint senderRemote = new IPEndPoint(IPAddress.Any, 0);
try
{
// Read data from the client socket.
int bytesRead = sock.EndReceiveFrom(ar, ref senderRemote);
if (bytesRead <= 0)
{
//handle disconnect logic
}
else
{
//handle the message received
}
}
catch (SocketException se)
{
Console.WriteLine(se.ToString());
}
// Signal the read thread to continue
readDone.Set();
}
Two exceptions are thrown, one of which I believe is being caught:
Exception thrown: 'System.Net.Sockets.SocketException' in System.dll
System.Net.Sockets.SocketException (0x80004005): An existing connection was forcibly closed by the remote host
at System.Net.Sockets.Socket.EndReceiveFrom(IAsyncResult asyncResult, EndPoint& endPoint)
at CardCatacombs.Utilities.Networking.UDPNetworkConnection.readCallback(IAsyncResult ar) in C:\Users\kayas\Desktop\Practicum\Source\CardCatacombs\CardCatacombs\Utilities\Networking\UDPNetworkConnection.cs:line 424
Exception thrown: 'System.Net.Sockets.SocketException' in System.dll
System.Net.Sockets.SocketException (0x80004005): An existing connection was forcibly closed by the remote host
at System.Net.Sockets.Socket.DoBeginReceiveFrom(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags, EndPoint endPointSnapshot, SocketAddress socketAddress, OverlappedAsyncResult asyncResult)
I would like to be able to cleanly handle a client crash and continue running since there are other clients connected to the server.

From this forum thread, it seems that the UDP socket is also receiving ICMP messages and throwing exceptions when they are received. If the port is no longer listening (after the hard crash), the ICMP message causes the 'forcibly closed' exception.
If not wanted, this exception can be disabled using the following code when creating the UdpClient, explained in the above post:
public const int SIO_UDP_CONNRESET = -1744830452;
var client = new UdpClient(endpoint);
client.Client.IOControl(
(IOControlCode)SIO_UDP_CONNRESET,
new byte[] { 0, 0, 0, 0 },
null
);
For dotnet core users, since this "Socket.IOControl" is a Windows-specific control code, other platforms are not supported, we will get the below exception:
Unhandled exception. System.PlatformNotSupportedException: Socket.IOControl handles Windows-specific control codes and is not supported on this platform.
For better compatibility, we should check the current platform:
using System.Runtime.InterosServices;
if (RuntimeInformation.IsOSPlatform(OSPlatform.Windows))
{
//...
}

System.ObjectDisposedException throwed on WebSocket communication

I'm trying to communicate between a web browser client and an ASP.NET server using WebSockets.
I make a set of requests, of different sizes and with some seconds of elapsed time between each of them. The three first ones pass correctly, but a precise one, with nothing in particular from the other, close the WebSocket connection, throw an exception on server side.
The error message and stack trace of this exception look like this :
FATAL ERROR: Cannot access a disposed object.
Object name: 'System.Web.WebSockets.AspNetWebSocket'.
at System.Web.WebSockets.AspNetWebSocket.ThrowIfDisposed()
at System.Web.WebSockets.AspNetWebSocket.SendAsyncImpl(ArraySegment 1 buffer, WebSocketMessageType messageType, Boolean endOfMessage, CancellationToken cancellationToken, Boolean performValidation)
at System.Web.WebSockets.AspNetWebSocket.SendAsync(ArraySegment 1 buffer, WebSocketMessageType messageType, Boolean endOfMessage, CancellationToken cancellationToken)
at [my code path here...]
It may be a threading problem, because I'm using async methods everywhere from functions that communicate with websockets.
I know the exception is throwed from this code (at socket.SendAsync):
public class SocketTranslater
{
private WebSocket socket;
private JavaScriptSerializer serializer;
// [...]
private ArraySegment<byte> Encode(Object data)
{
string json = serializer.Serialize(data);
return (new ArraySegment<byte>(Encoding.UTF8.GetBytes(json)));
}
public async Task Send(Object packet)
{
ArraySegment<byte> encoded = this.Encode(packet);
await socket.SendAsync(encoded, WebSocketMessageType.Text, true, CancellationToken.None);
}
}
The socket is created from another class :
public class EventSender : IHttpHandler
{
private static List<SocketTranslater> socketTranslaters = new List<SocketTranslater>();
public void ProcessRequest(HttpContext context)
{
this.id = context.Request.UserHostName + ":" + context.Request.UserAgent;
if (context.IsWebSocketRequest)
{
context.AcceptWebSocketRequest(ProcessSocket);
}
}
private async Task ManageSocket(AspNetWebSocketContext context)
{
WebSocket socket = context.WebSocket;
SocketTranslater translater = new SocketTranslater(socket);
translaters.add(translater);
while (true)
{
// Manage messages, using the other class to send responses.
translater.Send(/* Any struct here */);
}
}
Unhopefully, my project is too big to put all the code here.
Any idea of error source, or additional information that you would require ?
UPDATE:
After some more tests, I don't have this exception from time to time. I always do the same thing, but the server seems to have a random comportment.
That makes my problem even weirder...

Finally, after some more tests and interesting questions and answer from here (like this one), I understood:
My problem was that I was stocking WebSockets in a Dictionary linked with hostnames. So on the first connection of a client on my server, everything worked correctly. But if I refresh the page, the websocket was closed by the server (because there was no chance to use it again) et another one was created.
But as I used the same key for both sockets, the deprecated and the new one, I was trying to answer to the new client with the previous socket, that was closed. (disposed = closed for ASP.NET).
So the only thing that I had to do is to remove a socket from the list on the client disconnection (at the end of ManageSocket). And forbid a client to connect twice with the same hostname.
I didn't mention the part where I was linking sockets with hostnames, so I admit you couldn't really help me... I apologize.

AggregateException not being caught?

I'm querying a remote server and sometimes get an AggregateException. This is fairly rare and I know how to get around when this happens, but for some reason, it's not entering the catch block whenever the exception is thrown.
This is the code part for the catch block:
try
{
using (Stream stream = await MyQuery(parameters))
using (StreamReader reader = new StreamReader(stream))
{
string content = reader.ReadToEnd();
return content;
}
}
catch (AggregateException exception)
{
exception.Handle((innerException) =>
{
if (innerException is IOException && innerException.InnerException is SocketException)
{
DoSomething();
return true;
}
return false;
});
}
And this is the exception message I'm getting:
System.AggregateException: One or more errors occurred. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
--- End of inner exception stack trace ---
I'm assuming those --> arrows indicate that it's an inner exception right?
So if it's IOException -> SocketException, why is DoSomething() never called?

I suspect you're not actually seeing an AggregateException at this point. There's nothing in the code you have that is doing parallel operations.
If that's correct, you should be able to do something like this:
try
{
using (Stream stream = await MyQuery(parameters))
using (StreamReader reader = new StreamReader(stream))
{
string content = reader.ReadToEnd();
return content;
}
}
catch (IOException exception)
{
if (exception.InnerException is SocketException)
DoSomething();
else
throw;
}

No matter what I try: The I/O operation has been aborted because of either a thread exit or an application request

I try to build a simple async net tcp wcf tool which will open connection, send command, receive answer (a List with 0-10 string sentences), close connection.
The problem is, I get on (self-hosted) service side always - no matter what I try - "The I/O operation has been aborted because of either a thread exit or an application request", on client side of course the corresponding errors like "Existing connection was closed by remote host" and timeouts and so on.
I tried alot for the past days but I can't get rid of it.
Client Side (running on .NET 4.0, called around once a sec):
void callservice(string mykey) {
ServiceReference1.Client c = new ServiceReference1.Client();
c.GetDataCompleted += c_GetDataCompleted;
try {
c.GetDataAsync(mykey);
}
catch (FaultException aa)
{
c.Abort();
}
}
private void c_GetDataCompleted(object sender, ServiceReference1.GetDataCompletedEventArgs e)
{
ServiceReference1.Client c = (ServiceReference1.Client)sender;
c.GetDataCompleted -= c_GetDataCompleted;
try
{
if (e.Result != null && e.Result.Length > 0)
{
... }
c.Close();
}
catch (Exception) {
c.Abort();
}
}
Server Side (running on .NET4.5):
[ServiceBehavior(ConcurrencyMode=ConcurrencyMode.Multiple,
InstanceContextMode=InstanceContextMode.PerCall,IncludeExceptionDetailInFaults=true)]
public class Service1 : IMyService
{
public async Task<List<string>> GetData(string whatkey)
{
List<string> mydatalist = new List<string>();
mydatalist= await Task.Run<List<string>>(() =>
{
...
});
return mydatalist;
}
What is going wrong there? Could it be that it is something not having to do with WCF at all? What could it be?
Server Side Exception:
System.Net.Sockets.SocketException, System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089
The I/O operation has been aborted because of either a thread exit or an application request
at System.ServiceModel.Channels.SocketConnection.HandleReceiveAsyncCompleted()
at System.ServiceModel.Channels.SocketConnection.OnReceiveAsync(Object sender, SocketAsyncEventArgs eventArgs)
at System.Net.Sockets.SocketAsyncEventArgs.FinishOperationAsyncFailure(SocketError socketError, Int32 bytesTransferred, SocketFlags flags)
at System.Net.Sockets.SocketAsyncEventArgs.CompletionPortCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped)
at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)
System.Net.Sockets.SocketException (0x80004005): The I/O operation has been aborted because of either a thread exit or an application request
3E3
One more interesting fact:
SVCLogs show me that the I/O Exeption occurs after a timespan I can define in the
<connectionPoolSettings groupName="default" leaseTimeout="00:03:00"
idleTimeout="00:02:39" maxOutboundConnectionsPerEndpoint="20" />
settings.
In this example it will occur the first time after 00:02:39.
My interpretation: It closes open connections due to the settings there and that causes the Exception since the ReceiveAsync operation i.ex. was still open.
My question is so far why does client.close() not close it completely and why isn't it finished yet when it is calling the c_getdatacompleted-event? Why does the operation "hang out" for 02:39 minutes and does not come to an end?
(If I would not force the close down via the connectionpool settings I end up with hundreds of open operations if I use netstat i.ex. to display)

Async WCF operations (AsyncPattern=true) are implemented with the Asynchronous Programming Model. That is, you implement an operation ("Operation") with two asynchronous operations ("BeginOperation" and "EndOeration"). The client can wrap those operations with a Task (presumably with the FromAsync overload)
For example:
[ServiceContract]
public interface ISampleTaskAsync
{
[OperationContract(AsyncPattern = true)]
IAsyncResult BeginDoWork(int count, AsyncCallback callback, object state);
int EndDoWork(IAsyncResult result);
}
The WCF contract does not return a Task<T>
Then, on the client you could do something like:
var proxy = new Services.SampleTaskAsyncClient();
object state = "This can be whatever you want it to be";
var task = Task<int>.Factory.FromAsync(proxy.BeginDoWork,
proxy.EndDoWork, 10, state);
For more information see:
http://msdn.microsoft.com/en-us/library/system.servicemodel.operationcontractattribute.asyncpattern.aspx
http://blogs.msdn.com/b/rjacobs/archive/2011/06/30/how-to-implement-a-wcf-asynchronous-service-operation-with-task-lt-t-gt.aspx
If you want to use Task<T>, I believe you don't need AsyncPattern=true.